KDB/q count number of rows group by value of a column - kdb

I have a table like following
tickTime secid ltp
1.28E+18 37991 7350
1.28E+18 38596 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 38596 7350
1.28E+18 37986 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 37981 7350
1.28E+18 45402 7350
1.28E+18 45402 7350
1.28E+18 37981 7350
How can I do a count of no of rows for each secid in Q ?
Something like select count(*) group by secid in sql?

select count i by secid from tablename
This is how you create a count column. It will return the secid's and the number of times they occur in a table. Could rename the count column to "counts" as follows:
select counts:count i by secid from tablename

Related

How to search text in specific documents and not all documents in mongodb?

First I created index for my data:
db.stores.find( { $text: { $search: "java coffee shop" } } )
and here is the code for searching text inside the indexed field.
db.stores.find( { $text: { $search: "shop" } } )
but the problem is it searches the whole documents in my collection for the word "shop", and I want to limit its explore to some specific documents based on for example their id.
In other words, I want to just search the word "shop" through documents in my collection that
the value of their Occupation field is equal to "Z1";
You can just add the other field like this:
db.stores.find( { occupation: "Z1", $text: { $search: "shop" } } )

MongoDB UpdateMany

I need to update various fields in mongo creating a new field. Example:
db.paymentTransaction.updateOne(
{status: "PARTIALLY_REFUNDED"},
{amount: EXISTS}
{
$set: {
"amountRefundRemaining": {
FIELD_B - FIELD_A
}
}
}
)
I need to create the field amountRefundRemaining. But I need to fill this new field with the result of the substraction of Field_B minus Field_A. But I only need to do this where status=PARTIALLY_REFUNDED and amount exists
Im using mongoDB 4.4.
Any ideias?
Query
if status=PARTIALLY_REFUNDED and amount exists
add field amountRefundRemaining with value FIELD_B - FIELD_A
*pipeline update requires MongoDB >= 4.2
PlayMongo
update(
{"$and":
[{"status": {"$eq": "PARTIALLY_REFUNDED"}}, {"amount": {"$exists": true}}]},
[{"$set":
{"amountRefundRemaining": {"$subtract": ["$FIELD_B", "$FIELD_A"]}}}],
{"multi": true})

Is there a way to update all documents in a collection with a random value in MongoDB?

I want to increase a field in all documents with a random value and return the sum of all the increments.
I made it work by iterating through all collection but obviously it is very slow.
const earnMoneyForUsers = async () => {
let usersCursor = await User.find()
.select({ name: 1, weeklyEarnedMoney: 1, totalEarnedMoney: 1 })
.cursor()
let totalEarnedMoney = 0
await usersCursor.eachAsync((user) => {
let earnedMoneyByUser = Math.floor(Math.random() * 100)
let collectedAmount = earnedMoneyByUser * 0.02
user.weeklyEarnedMoney += earnedMoneyByUser - collectedAmount
user.totalEarnedMoney += earnedMoneyByUser - collectedAmount
totalEarnedMoney += earnedMoneyByUser - collectedAmount
user.save()
})
return totalEarnedMoney
}
I tried updateMany also but it updates all values with same random value and I don't know how I can sum the increment?
const earnMoneyForUsers = async () => {
await User.updateMany(
{},
{ $inc: { weeklyEarnedMoney: Math.floor(Math.random() * 100) } }
)
return ?
}
you can use $rand aggregation operator, and do a pipeline update like the bellow, doing all the calculations in the database, without sending data to the application.
The bellow is a way to re-write your code in MQL
let earnedMoneyByUser = Math.floor(Math.random() * 100)
let collectedAmount = earnedMoneyByUser * 0.02
user.weeklyEarnedMoney += earnedMoneyByUser - collectedAmount
user.totalEarnedMoney += earnedMoneyByUser - collectedAmount
PlayMongo
update(
{},
[{"$set":
{"earnedMoneyByUser": {"$toInt": {"$multiply": [{"$rand": {}}, 100]}}}},
{"$set":
{"collectedAmount": {"$multiply": ["$earnedMoneyByUser", 0.02]}}},
{"$set":
{"weeklyEarnedMoney":
{"$add":
["$weeklyEarnedMoney",
{"$subtract": ["$earnedMoneyByUser", "$collectedAmount"]}]}}},
{"$set":
{"totalEarnedMoney":
{"$add":
["$weeklyEarnedMoney",
{"$subtract": ["$earnedMoneyByUser", "$collectedAmount"]}]}}}],
{"multi": true})

multiplication and group by in mongodb

I have a collection as follow in mongodb :
{
"_id" : ObjectId("54901212f315dce7077204af"),
"Date" : ISODate("2014-10-20T04:00:00.000Z"),
"Type" : "Twitter",
"Entities" : [
{
"ID" : 2,
"Name" : "test1",
"Sentiment" : {
"Value" : 20,
"Neutral" : 1
},
{
"ID" : 1,
"Name" : "test1",
"Sentiment" : {
"Value" : 1,
"Neutral" : 1
}
},
{
"ID" : 3,
"Name" : "test1",
"Sentiment" : {
"Value" : 2,
"Neutral" : 1
}
]
}
and I have couple of them , for example in date 2014-10-20 you might find 5 tweets each of which have different value for sentiment, now what I want to do is to group by date and then get the sum of sentiment value for each date multiply it by the number of collections for each date, for example if we have 2 collections in 2014-10-20 with sentiment values of 20,1,2 like the collection showed above and just 5 for another collection then the value for 2014-10-20 is (20+1+2+5)3(because this tweet is repeated for 3 entites) 2(because we have 2 tweets document in this date)=168 , if I do not consider frequency of collections my code works well as follow :
DBObject unwind = new BasicDBObject("$unwind", "$Entities"); // "$unwind" converts object with array into many duplicate objects, each with one from array
collectionG = db.getCollection("GraphDataCollection");
DBObject groupFields = new BasicDBObject( "_id", "$Date");
groupFields.put("value", new BasicDBObject( "$sum", "$Entities.Sentiment.Value"));
DBObject groupBy = new BasicDBObject("$group", groupFields );
DBObject sort = new BasicDBObject("$sort", new BasicDBObject("Date", 1));
stages.add(unwind);
stages.add(groupBy);
DBObject project = new BasicDBObject("_id",0);
project.put("Date","$_id");
project.put("value",1);
stages.add(new BasicDBObject("$project",project));
stages.add(sort);
AggregationOutput output = collectionG.aggregate(stages);
Now the result for for example 2014-10-20 returns 28 but I want 168
can anyone help me ?
Update : the last version of the code that I used is as follow:
DBCollection collectionG;
collectionG = db.getCollection("GraphDataCollection");
List<DBObject> stages = new ArrayList<DBObject>();
ArrayList<DBObject> andArray = null;
DBObject groupFields = new BasicDBObject( "_id", "$_id");
groupFields.put("value", new BasicDBObject( "$sum", "$Entities.Sentiment.Value"));
groupFields.put("date", new BasicDBObject( "$first", "$Date"));
DBObject groupBy = new BasicDBObject("$group", groupFields );
stages.add(groupBy);
DBObject groupByDate = new BasicDBObject( "_id", "$date");
groupByDate.put("value",new BasicDBObject("$sum","$value"));
groupByDate.put("count",new BasicDBObject("$sum",1));
DBObject dtGrp = new BasicDBObject("$group", groupByDate );
stages.add(dtGrp);
DBObject project = new BasicDBObject("_id",1);
project.put("value",new BasicDBObject("$multiply",
new Object[]{"$value","$count"}));
stages.add(new BasicDBObject("$project",project));
AggregationOutput output = collectionG.aggregate(stages);
System.out.println(output.results());
Unwind Entities:
DBObject unwind = new BasicDBObject("$unwind", "$Entities");
stages.add(unwind);
Group by _id to find the sum of all the Entities sentiment values per document.
DBObject groupFields = new BasicDBObject( "_id", "$_id");
groupFields.put("value", new BasicDBObject( "$sum", "$Entities.Sentiment.Value"));
groupFields.put("date", new BasicDBObject( "$first", "$Date"));
DBObject groupBy = new BasicDBObject("$group", groupFields );
stages.add(groupBy);
Group by Date now, to get the sum of total Entities Value, and the count of documents per group.
DBObject groupByDate = new BasicDBObject( "_id", "$date");
groupByDate.put("value",new BasicDBObject("$sum","$value"));
groupByDate.put("count",new BasicDBObject("$sum",1));
DBObject dtGrp = new BasicDBObject("$group", groupByDate );
stages.add(dtGrp);
Project value as the multiplicative result of count and value, for each group.
DBObject project = new BasicDBObject("_id",1);
project.put("value",new BasicDBObject("$multiply",
new Object[]{"$value","$count"}));
stages.add(new BasicDBObject("$project",project));
In case your dates differ by milliseconds, you need to group by the date, year and month together, in the second group stage and add a sort stage if necessary.

Nested query with aggregation in Mongo

I have a document in MongoDB:
{
"_id" : ObjectId("111111111111111111111111"),
"taskName" : "scan",
"nMapRun" : {
...
"hosts" : {
...
"distance" : {
"value" : "1"
},..
}
I'm interested in the field: nMapRun.hosts.distance.value
How do I get ten maximum values ​​of the field .
Could you give an example of a Java?
The aggregation operation in shell:
db.collection.aggregate([
{$sort:{"nMapRun.hosts.distance.value":-1}},
{$limit:10},
{$group:{"_id":null,"values":{$push:"$nMapRun.hosts.distance.value"}}},
{$project:{"_id":0,"values":1}}
])
You need to build the corresponding DBObjects for each stage as below:
DBObject sort = new BasicDBObject("$sort",
new BasicDBObject("nMapRun.hosts.distance.value", -1));
DBObject limit = new BasicDBObject("$limit", 10);
DBObject groupFields = new BasicDBObject( "_id", null);
groupFields.put("values",
new BasicDBObject( "$push","$nMapRun.hosts.distance.value"));
DBObject group = new BasicDBObject("$group", groupFields);
DBObject fields = new BasicDBObject("values", 1);
fields.put("_id", 0);
DBObject project = new BasicDBObject("$project", fields );
Running the aggregation pipeline:
List<DBObject> pipeline = Arrays.asList(sort, limit, group, project);
AggregationOutput output = coll.aggregate(pipeline);
output.results().forEach(i -> System.out.println(i));