I installed Birt 4.3 (latest) as Eclipse plugin.
JDBC as data source works perfectly for me. I define a param in query. I can define a param in data set via Report Design and link it to a report param.
But I have multiple issues with MongoDB
1) When I open "edit data set" dialog I cannot find the menu "Parameters" in left side. I only see
Data Source
Query
Output Columns
Computed Columns
Filters
properties Binding
Setting
Review Results
No Parameter any more. I can see it in JDBC not in MongoDB
Could anyone please how to define a parameter in data set while Data source is MongoDB? How to link the param to the param of report?
2) In MongoDB (Birt) how to define a parameter in the expression of "Run Database Command" or in query
In JDBC we can use "?" as a param holder like
select * where id=?
param will replace "?"
For MongoDB
{
runCommand : {
aggregate : COLLECTION_NAME,
.....
}
}
COLLECTION_NAME is vary. How can I represent it?
Any help will be appreciated.
Thanks
MongoDB expression syntax itself does not support parameters, thus the MongoDB ODA connector does not support data set parameters directly. You can use BIRT scripting to reference a BIRT report parameter and change the runtime value of the "Command expression" data set property.
You can identify the system connection when you make a MongoDB data set using Eclipse. This is in the MongoDB Collection area of the data set dialog.
Check http://www.eclipse.org/forums/index.php/t/628348/ for more information.
EXAMPLE
You can use script to set the query using a beforeOpen script in the dataset such as:
this.queryText ='{ "findQueryExpr" : "{ pop: { $gte: 20000 } }" , "operationType" : "FIND" , "collectionName" : "zipcode" , "selectedFields" : [ "_id" , "city", "pop"]}'
The fields and collection name need to match the pre-configured dataset.
Replace the 20000 with your parameter value. For example, using a parameter named "Population" which takes an integer, you can update your query at run-time with the following script:
this.queryText ='{ "findQueryExpr" : "{ pop: { $gte: '+params["Population"].value+' } }" , "operationType" : "FIND" , "collectionName" : "zipcode" , "selectedFields" : [ "_id" , "city", "pop"]}'
Related
Okay, I'm a SQL Server based DBA, but there's a biz-critical app that uses MongoDB as a core component, and the problem is that it's grown too large. (The "large-ness" isn't critical yet, I'm trying to be proactive!)
Specifically, it has a "message log" collection which is over 400 GB, where the date-stamp is actually stored as a 2-element array [Int64, Int32], the 0th element being some measure of time (and the 1th element is just always '0').
So for example, a document:
{
"_id" : ObjectId("55ef63618c782e0afcf346cf"),
"CertNumber" : null,
"MachineName" : "WORKERBEE1",
"DateTime" : [
NumberLong(635773487051900000),
0
],
"Message" : "Waited 00:00:30.0013381 to get queue lock: Specs to verify",
"ScopeStart" : false
}
And just because 2 is better than 1, another example document:
{
"_id" : ObjectId("55ef63618c782e0afcf323be"),
"CertNumber" : null,
"MachineName" : "WORKERBEE2",
"DateTime" : [
NumberLong(635773487056430453),
0
],
"Message" : "Waited 00:00:30.0012345 to get queue lock: Specs to verify",
"ScopeStart" : false
}
I need to figure out two things:
What the heck does that "DateTime" really mean? It's not Unix Epoch time (seconds nor milliseconds); and even if I strip off the trailing 0's, it represents (in millis) 6/20/2171, so, unless we're building a time machine here, it makes no sense. If I strip off the last 6 digits, it means 2/23/1990, but even that doesn't seem likely, as this application has only existed since the early 2000's. (AFAIK)
Assuming we figure out #1, can we use some kind of command to remove (delete) all documents in the collection that are older than, say, 1/1/2016?
Again, I'm a SQL guy, so try to explain using analogs in that vein, e.g. "this is like your WHERE clause" and such.
PS: Yes, I read thru questions such as Find objects between two dates MongoDB and How do I convert a property in MongoDB from text to date type? , but so far nothing has jumped out at me.
I want to use the templating feature in Grafana with elastic search to create a set of 'dynamic' terms ("application")
To get the ist of terms from elasticsearch I'm useing:
{
"aggs" :
{
"applications" : {
"terms" : { "field" : "businessTransactions.application" }
}
}
}
When I use that query in the Templating Query variable settings as query Grafana tells me: "Template variables could not be initialized: Cannot read property 'then' of undefined"
I'm using grafana 3.1.0beta1
Maybe I'm completely off, but how would someone use a query to get different terms of a field as a template variable from elasticsearch?
Thanks!
First question: which version of Grafana are you using? Sorry just re-read and saw the answer, which is 3.1.0beta1.
The below works for me on 3.1.0 (not beta).
Second question: did you see this page: http://docs.grafana.org/datasources/elasticsearch/
Templating
The Elasticsearch datasource supports two types of queries
you can use to fill template variables with values.
Possible values for a field
{"find": "terms", "field": "#hostname"}
Fields filtered by type
{"find": "fields", "type": "string"}
Fields filtered by type, with filter {"find": "fields", "type": "string", "query": <lucene query>}
Multi format / All format
We are investigating whether Hive will allow us to run some SQL-like queries on
mongo style dynamic schema as a precursor to our map-reduce jobs.
The data comes in the form of several TiB of BSON files; each of the files contains
JSON "samples". An example sample is given as such:
{
"_id" : "SomeGUID",
"SomeScanner" :
{
"B64LR" : 22,
"Version" : 192565886128245
},
"Parser" :
{
"Size" : 73728,
"Headers" :
[
{
"VAddr" : 4096,
"VSize" : 7924.
. . . etc. . . .
As a dynamic schema, only a few of the fields are guaranteed to exist.
We would like to be able to run a query against an input set that may be something
like
SomeScanner.Parser.Headers.VSize > 9000
Having looked up the table-mapping, I'm not sure whether this is do-able with Hive . . . how would one map a column that may or may not be there . . . not to mention that there are about 2k-3k query-able values in a typical sample.
Hence, my questions to the Experts:
Can Hive build a dynamic schema from the data it encounters?
How can one go about building a Hive table with ~3k columns?
Is there a better way?
Appreciated, as always.
OK--with much ado, I can now answer my own questions.
Can Hive build a dynamic schema from the data it encounters?
A: No. However, an excellent tool for this exists. q.v., inf.
How can one go about building a Hive table w/~3K columns
A: Ibidem
Is there a better way?
A: Not that I found; but, with some help, it isn't too difficult.
First, a shout out to Michael Peterson at http://thornydev.blogspot.com/2013/07/querying-json-records-via-hive.html, whose blog post served as the toe-hold to figure all of this out.
Definitely check it out if you're starting out w/Hive.
Now, Hive cannot natively import a JSON document and deduce a schema from it . . . however, Michael Peterson has developed tool that does: https://github.com/midpeter444/hive-json-schema
Some caveats with it:
* Empty arrays and structs are not handled, so remove (or populate) them. Otherwise, things like { "something" : {} } or {"somethingelse": []} will throw errors.
If any field has the name "function", it will have to be re-named prior to executing the CREATE TABLE statement. E.g., the following code would throw an error: 1{ "someThing": { "thisIsOK": "true", "function": "thatThrowsAnError" } }`
Presumably, this is because "function" is an Hive keyword.
And, with dynamic schema in general, I have not found a way to handle a nested leading underscore name even if the schema is valid: { "somethings": { "_someVal": "123", "otherVal": "456" } } will fail.
For the common MongoDB "ID" field, this is map-able with the following addition: with serdeproperties("mapping.id" = "_id"), which appears to be similar to a macro-substitution.
Serialization/De-Serialization for JSON can be achieved with https://github.com/rcongiu/Hive-JSON-Serde by adding the following: ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
N.B., the JsonSerDe JAR must have been added to .hiverc or "add jar"'d into Hive to be used.
Thus, the schema:
CREATE TABLE samplesJSON
( id string,
. . . rest of huge schema . . . . )
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH serdeproperties("mapping.id" = "_id");
The JSON data can be loaded into the table with a command along the lines of:
LOAD DATA LOCAL INPATH '/tmp/samples.json' OVERWRITE INTO TABLE samplesJSON;
Finally, queries are actually intuitive straight-forward. Using the above example from the original question:
hive> select id, somescanner.parser.headers.vaddr from samplesjson;
OK
id vaddr
119 [4096,53248,57344]
Is there a possibility to use the full text search of mongoDB with the haskell driver?
I found the 'runCommand' in the haskell API, but it expects a Document as parameter. That's fine for all the other commands that mongodb can run, but the syntax for a text command is:
db.collection.runCommand( "text", {search : "something"})
So I don't know how I'll get the "text" as first parameter in front of the Document.
Thanks
The text-command can be written in another structure:
{ text: your_collection
, search: your_text
, filter: your_filter
, limit: your_limit
, project: your_projection
}
I had my suspicion, since all "runCommand"-action have the same structure. So I tried to apply that structure to the text-command - but without success. Then I remembered that aggregate has another structure too and tried that, but that didn't work either. Finally, I found the answer in a google group entry of the Java driver.
I can't find how to format my datas in Erlang and choose the encoding style provided by Erlmongo (default or mochijson) to be able to save datas as an Array in a Mongo document.
Example of a Mongo Document containing an array :
{
"_id" : MyId,
datas : [0,1,2,3]
}
This type of document is correct in Mongo format.
But how to create such a document with the erlmongo driver ?
I believe that you want to build an erlang property list. Your example JSON document in mongo might look like this as an erlang proplist:
[
{id, MyId},
{datas, [0, 1, 2, 3]}
]
Try this out and let me know if it works. There are a couple different mongo drivers for erlang, and I think I've only used emongo.
Also, some advice that you didn't ask for: "data" is already the plural form of "datum". You don't add an "s".
Hope this helps.
-tjw