Logging mongodb query in java - mongodb

I am using mongo-java-driver-2.9.1 for interacting with mongodb, I want to log the query that are fired on to the mongodb server. e.g. In java for inserting the document this is the code that I write
DBCollection coll = db.getCollection("mycollection");
BasicDBObject doc = new BasicDBObject("name", "MongoDB")
.append("type", "database")
.append("count", 1);
coll.insert(doc);
for this, equivalent code in "mongo" client for inserting document in mongodb is
db.mycollection.insert({
"name" : "MongoDB",
"type" : "database",
"count" : 1
})
I want to log this second code, is there any way to do it?

I think the MongoDB Java driver has not logging support so you have to write your logging Message by your own. Here an Example:
DBCollection coll = db.getCollection("mycollection");
BasicDBObject doc = new BasicDBObject("name", "MongoDB")
.append("type", "database")
.append("count", 1);
WriteResult insert = coll.insert(doc);
String msg = "";
if(insert.getError() == null){
msg = "insert into: " + collection.toString() +" ; Object " + q.toString());
//log the message
} else {
msg = "ERROR by insert into: " + collection.toString() +" ; Object " + q.toString());
msg = msg + " Error message: " + insert.getError();
}
//log the message

Related

Flink SQL Client connect to secured kafka cluster

I want to execute a query on Flink SQL Table backed by kafka topic of secured kafka cluster. I'm able to execute the query programmatically but unable to do the same through Flink SQL client. I'm not sure on how to pass JAAS config (java.security.auth.login.config) and other system properties through Flink SQL client.
Flink SQL query programmatically
private static void simpleExec_auth() {
// Create the execution environment.
final EnvironmentSettings settings = EnvironmentSettings.newInstance()
.inStreamingMode()
.withBuiltInCatalogName(
"default_catalog")
.withBuiltInDatabaseName(
"default_database")
.build();
System.setProperty("java.security.auth.login.config","client_jaas.conf");
System.setProperty("sun.security.jgss.native", "true");
System.setProperty("sun.security.jgss.lib", "/usr/libexec/libgsswrap.so");
System.setProperty("javax.security.auth.useSubjectCredsOnly","false");
TableEnvironment tableEnvironment = TableEnvironment.create(settings);
String createQuery = "CREATE TABLE test_flink11 ( " + "`keyid` STRING, " + "`id` STRING, "
+ "`name` STRING, " + "`age` INT, " + "`color` STRING, " + "`rowtime` TIMESTAMP(3) METADATA FROM 'timestamp', " + "`proctime` AS PROCTIME(), " + "`address` STRING) " + "WITH ( "
+ "'connector' = 'kafka', "
+ "'topic' = 'test_flink10', "
+ "'scan.startup.mode' = 'latest-offset', "
+ "'properties.bootstrap.servers' = 'kafka01.nyc.com:9092', "
+ "'value.format' = 'avro-confluent', "
+ "'key.format' = 'avro-confluent', "
+ "'key.fields' = 'keyid', "
+ "'value.fields-include' = 'EXCEPT_KEY', "
+ "'properties.security.protocol' = 'SASL_PLAINTEXT', 'properties.sasl.kerberos.service.name' = 'kafka', 'properties.sasl.kerberos.kinit.cmd' = '/usr/local/bin/skinit --quiet', 'properties.sasl.mechanism' = 'GSSAPI', "
+ "'key.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', "
+ "'key.avro-confluent.schema-registry.subject' = 'test_flink6', "
+ "'value.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', "
+ "'value.avro-confluent.schema-registry.subject' = 'test_flink4')";
System.out.println(createQuery);
tableEnvironment.executeSql(createQuery);
TableResult result = tableEnvironment
.executeSql("SELECT name,rowtime FROM test_flink11");
result.print();
}
This is working fine.
Flink SQL query through SQL client
Running this giving the following error.
Flink SQL> CREATE TABLE test_flink11 (`keyid` STRING,`id` STRING,`name` STRING,`address` STRING,`age` INT,`color` STRING) WITH('connector' = 'kafka', 'topic' = 'test_flink10','scan.startup.mode' = 'earliest-offset','properties.bootstrap.servers' = 'kafka01.nyc.com:9092','value.format' = 'avro-confluent','key.format' = 'avro-confluent','key.fields' = 'keyid', 'value.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', 'value.avro-confluent.schema-registry.subject' = 'test_flink4', 'value.fields-include' = 'EXCEPT_KEY', 'key.avro-confluent.schema-registry.url' = 'http://kafka-schema-registry:5037', 'key.avro-confluent.schema-registry.subject' = 'test_flink6', 'properties.security.protocol' = 'SASL_PLAINTEXT', 'properties.sasl.kerberos.service.name' = 'kafka', 'properties.sasl.kerberos.kinit.cmd' = '/usr/local/bin/skinit --quiet', 'properties.sasl.mechanism' = 'GSSAPI');
Flink SQL> select * from test_flink11;
[ERROR] Could not execute SQL statement. Reason:
java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is /tmp/jaas-6309821891889949793.conf
There is nothing in /tmp/jaas-6309821891889949793.conf except the following comment
# We are using this file as an workaround for the Kafka and ZK SASL implementation
# since they explicitly look for java.security.auth.login.config property
# Please do not edit/delete this file - See FLINK-3929
SQL client run command
bin/sql-client.sh embedded --jar flink-sql-connector-kafka_2.11-1.12.0.jar --jar flink-sql-avro-confluent-registry-1.12.0.jar
Flink cluster command
bin/start-cluster.sh
How to pass this java.security.auth.login.config and other system properties (that I'm setting in the above java code snippet), for SQL client?
flink-conf.yaml
security.kerberos.login.use-ticket-cache: true
security.kerberos.login.principal: XXXXX#HADOOP.COM
security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /path/to/kafka.keytab
security.kerberos.login.principal: XXXX#HADOOP.COM
security.kerberos.login.contexts: Client,KafkaClient
I haven't really tested whether this solution is feasible, you can try it out, hope it will help you.

set shell mode for mongodb [duplicate]

I am using MongoDB Driver Java API to convert BSON to JSON.
I have test code like this.
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output = " + doc.toJson());
The output is:
input = { "timestamp" : 1486064586641 }
output = { "timestamp" : { "$numberLong" : "1486064586641" } }
Is there an easy way to make the output look like the input?
BSON Documnet's toJson method supports only output to MongoDB Extended JSON (STRICT or SHELL format). If you want to have regular JSON, you can use com.mongodb.util.JSON class:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
System.out.println("output (SHELL) = " + doc.toJson(new JsonWriterSettings(JsonMode.SHELL)));
System.out.println("output (STRICT) = " + doc.toJson(new JsonWriterSettings(JsonMode.STRICT)));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc));
This will generate following output:
input = { "timestamp" : 1486064586641 }
output (SHELL) = { "timestamp" : NumberLong("1486064586641") }
output (STRICT) = { "timestamp" : { "$numberLong" : "1486064586641" } }
output (JSON) = { "timestamp" : 1486064586641}
Natalja's answer is excellent, but if you are using the Mongo Java driver 3.8.2 upwards you will notice some deprecation warnings. If you want the output to look like the input you can use RELAXED JsonWriterSettings mode.
Below you can see an example with the possible modes and how the JSON will looks like. There are also some deprecation warnings and alternatives to the deprecated code:
String input = "{ \"timestamp\" : 1486064586641 }";
org.bson.Document doc = org.bson.Document.parse(input);
System.out.println("input = " + input);
JsonWriterSettings shellMode = JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build();
System.out.println("output (SHELL) = " + doc.toJson(shellMode));
JsonWriterSettings strictMode = JsonWriterSettings.builder().outputMode(JsonMode.STRICT).build();
System.out.println("output (STRICT) = " + doc.toJson(strictMode)); // deprecated - use extended like below
JsonWriterSettings extendedMode = JsonWriterSettings.builder().outputMode(JsonMode.EXTENDED).build();
System.out.println("output (EXTENDED) = " + doc.toJson(extendedMode));
JsonWriterSettings relaxed = JsonWriterSettings.builder().outputMode(JsonMode.RELAXED).build();
System.out.println("output (RELAXED) = " + doc.toJson(relaxed));
System.out.println("output (JSON) = " + com.mongodb.util.JSON.serialize(doc)); // deprecated - use relaxed like above
Also note that the JsonWriterSettings constructor is deprecated and you can use as an alternative the builder method like e.g:
JsonWriterSettings.builder().outputMode(JsonMode.SHELL).build()

Getting Object tag value in AWS S3

I am using scala to get information about my objects that are in S3 I am intersted in getting each object I have inside S3 Bucket his tag value so far I have accomplished this code to get me information about my objects but did not succeeded in getting its tagging value: I used this code in scala:
def retrieveObjectTags(keyName: String): Unit ={
try {
println("Listing objects")
val req: ListObjectsV2Request =
new ListObjectsV2Request().withBucketName(bucketName).withMaxKeys(2)
var result: ListObjectsV2Result = null
do {
result = client.listObjectsV2(req)
for (objectSummary <- result.getObjectSummaries) {
println(
" - " + objectSummary.getKey + " " + "(size = " + objectSummary.getSize +
")")
println(objectSummary.getETag)
}
println("Next Continuation Token : " + result.getNextContinuationToken)
req.setContinuationToken(result.getNextContinuationToken)
} while (result.isTruncated == true);
}catch {
case ase: AmazonServiceException => {
println(
"Caught an AmazonServiceException, " + "which means your request made it " +
"to Amazon S3, but was rejected with an error response " +
"for some reason.")
println("Error Message: " + ase.getMessage)
println("HTTP Status Code: " + ase.getStatusCode)
println("AWS Error Code: " + ase.getErrorCode)
println("Error Type: " + ase.getErrorType)
println("Request ID: " + ase.getRequestId)
}
case ace: AmazonClientException => {
println(
"Caught an AmazonClientException, " + "which means the client encountered " +
"an internal error while trying to communicate" +
" with S3, " +
"such as not being able to access the network.")
println("Error Message: " + ace.getMessage)
}
}
// val getTaggingRequest = new GetObjectTaggingRequest(bucketName,keyName)
// var getTagResult = client.getObjectTagging(getTaggingRequest)
//println(getTaggingRequest)
var tag: Tag = new Tag()
println("tag name:" + tag.getValue)
}
as for the remarked lines I have encounter a problem with it, what other way I can use to solve this problem?

CQL SELECT fails with prepareStatement

In my sample app following snippet works fine.
SELECT_CQL = "SELECT * FROM " + STREAM_NAME_IN_CASSANDRA + " WHERE '" + CONTEXT_ID_COLUMN + "'=?";
Connection connection = getConnection();
statement = connection.prepareStatement(SELECT_CQL);
statement.setString(1, "123");
resultSet = statement.executeQuery();
But when I try to add another parameter to the where clause query returns nothing!!
SELECT_CQL = "SELECT * FROM " + STREAM_NAME_IN_CASSANDRA + " WHERE '" + CONTEXT_ID_COLUMN + "'=? AND '"+TIMESTAMP_COLUMN+"'=?";
Connection connection = getConnection();
statement = connection.prepareStatement(SELECT_CQL);
statement.setString(1, "123");
statement.setString(2, "1390996577514");
resultSet = statement.executeQuery();
When I try the exact query within cqlsh terminal, it works fine.
statement.setString(2, "1390996577514");
Double-check the datatype of your TIMESTAMP_COLUMN and make sure that it's a string. Otherwise you'll need to use the appropriate "set" method. Ex:
statement.setLong(2, 1390996577514L);

FileMaker 10 Pro Insert data by Asp.Net

I am trying to insert the data in FM but get the parse error serached number of fourms with no luck.
ERROR [HY000] [DataDirect][ODBC SequeLink driver][ODBC Socket][DataDirect][ODBC FileMaker driver][FileMaker]Parse Error in SQL
enter code here
StringBuilder sbAddBarcode = new StringBuilder();
sbAddBarcode.Append("insert into BarCode (PONumber,Description,Model,[Serial Number])");
sbAddBarcode.Append("values");
sbAddBarcode.Append(" ("+ barcode.PONumber + ",");
sbAddBarcode.Append(" '" + barcode.Description +"',");
sbAddBarcode.Append(" '" + barcode.ModelNumber +"')");
//sbAddBarcode.Append(" '" + barcode.SerialNumber +"')");
fmCommand = new OdbcCommand(sbAddBarcode.ToString(), fmcon);
fmCommand.CommandType = CommandType.Text;
fmCommand.Connection = fmcon;
try
{
fmcon.Open();
fmCommand.ExecuteNonQuery();
}
catch (OdbcException oe)
{
throw new Exception(oe.Message);
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
finally
{
fmcon.Close();
}
Echo your sbAddBarcode value. The SQL you're building does look to be invalid.