I use Doctrine 2 ODM\MongoDB (this and dependencies all latest from GIT) + PHP Mongo extension (v1.1.4) + Mongo (v1.8.3 x64 latest)
In several cases it happens that when I change an entity, it does not get updated
In an embedded entity I have a back reference to it's parent, to solve http://groups.google.com/group/doctrine-dev/browse_thread/thread/016422b2cc9dcec9?pli=1 (How to get parent document inside embedded document?) without using internal functionalities like UnitOfWork - this MAY or MAY NOT be relevant, I do not know... I would hope that not (my question whether it could went unanswered on the IRC so far)
However a single change in one of Character's embedded documents triggers these queries (should really be relevant only to it, there's been flush before it and a condition inside the logger on the flag that was set again right before the change)
Logger dump:
array(6) {
"update" => TRUE
"query" => array(1) {
"_id" => MongoId(1) {
"$id" => "4e3143be662c32400d000006" (24)
}
}
"newObj" => array(1) {
"$set" => array(2) {
"components.destructible.hp" => 99.28706276
"options.character" => array(3) {
"$ref" => "characters" (10)
"$id" => MongoId(1) { ... }
"$db" => "nuclearRain" (11)
}
}
}
"options" => array(0)
"db" => "someDB" (11)
"collection" => "characters" (10)
}
array(6) {
"update" => TRUE
"query" => array(1) {
"_id" => MongoId(1) {
"$id" => "4e553306662c32680800054b" (24)
}
}
"newObj" => array(1) {
"$set" => array(1) {
"createdBy" => array(3) {
"$ref" => "characters" (10)
"$id" => MongoId(1) { ... }
"$db" => "nuclearRain" (11)
}
}
}
"options" => array(0)
"db" => "someDB" (11)
"collection" => "worlds" (6)
}
array(6) {
"update" => TRUE
"query" => array(1) {
"_id" => MongoId(1) {
"$id" => "4e553306662c32680800054c" (24)
}
}
"newObj" => array(1) {
"$set" => array(1) {
"world" => array(3) {
"$ref" => "worlds" (6)
"$id" => MongoId(1) { ... }
"$db" => "nuclearRain" (11)
}
}
}
"options" => array(0)
"db" => "someDB" (11)
"collection" => "games" (5)
}
Initialization code: http://pastebin.com/1dKagqry
"components.destructible.hp" => 99.28706276 is the core part -> yet this change is never reflected in the DB
there is no php error/exception triggered
mongo's log shows no errors
generally the Mongo and Doctrine\ODM\Mongo runs fine, entities get persisted, updated, deleted...
Where can I search further or what can I do to solve this issue
Thank you
It is awkward, but the issue is that I wanted the EmbeddedDocuments indexed (like associative array)... this was however done on an actual Doctrine-maintained property, so it triggered re-pushing the EmbeddedDocument, discarding the changes
$friend = Zend_Registry::get('doctrine')->getDocumentManager()->createQueryBuilder('App\document\Message')->update()->field('unread')->set(TRUE)->field('viewer_id')->equals(10001)-> getQuery(array('multiple' => true))->execute();
Related
So I'm trying to use logstash to move data from MongoDB to elasticsearch. I don't want duplicates to be written, so I'm using the doc_as_upsert => true along with the document_id parameters in the output. This is my config file for logstash
input {
jdbc{
jdbc_driver_class => "com.dbschema.MongoJdbcDriver"
jdbc_driver_library => "/path/to/mongojdbc1.8.jar"
jdbc_user => ""
jdbc_password => ""
jdbc_connection_string => "jdbc:mongodb://127.0.0.1:27017/db1"
statement => "db1.coll1.find({ },{'_id': false})"
}
}
output {
elasticsearch {
hosts => ["http://127.0.0.1:9200"]
index => "test"
user => ""
password => ""
doc_as_upsert => true
document_id => "%{datetime}"
}
}
As you can see, I'm trying to use the datetime field of the MongoDB document (which is a string) as the document id for elasticsearch. But this is what a document inserted into Elasticsearch looks like:
{
"_index" : "test",
"_type" : "_doc",
"_id" : "%{datetime}",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2020-05-28T08:53:28.244Z",
"document" : {
# .. some fields ..
"datetime" : "2020-05-28 14:22:29.133363",
# .. some fields ..
},
"#version" : "1"
}
}
Instead of the value of the datetime field being used as the _id, the string %{datetime} is being used as the ID. How do I fix this?
The document_id field is not at the root level, so you need to change your syntax to the following:
document_id => "%{[document][datetime]}"
Below is my MongoDB collection
{
"_id" : ObjectId("5b2a2ee19d332d0118b26dfe"),
"Name" : "Rock",
"Job" : [
{
"Id" : ObjectId("5b2b93c63629d0271ce366ae"),
"JobName" : "abc",
"JobTrack" : [
"123"
]
}
]
}
I want to fetch both ObjectId values
my $cursor = $custColl->find(
{ 'Job.JobName' => "abc", 'Job.JobTrack' => "123" },
{ '_id' => 1, 'Job.Id' => 1 }
);
while ( my $next = $cursor->next ) {
my $CustomerId = "$next->{_id}";
my $JobId = "$next->{'Job.Id'}";
say "$CustomerId => $JobId\n";
}
The result I got from above code as follows
5b2a2ee19d332d0118b26dfe =>
With this code I'm not able to get $JobId.
Assuming the query finds a document, $next is a Perl data structure that resembles your original JSON:
{
'Name' => 'Rock',
'_id' => bless( {
'value' => '5b2a2ee19d332d0118b26dfe'
}, 'MongoDB::OID' ),
'Job' => [
{
'JobName' => 'abc',
'JobTrack' => [
'123'
],
'Id' => bless( {
'value' => '5b2b93c63629d0271ce366ae'
}, 'MongoDB::OID' )
}
]
}
To get the job ID, you need to dereference that structure using Perl syntax, not MongoDB syntax:
my $JobId = "$next->{Job}[0]{Id}";
You need to access a specific element of the Job array. Because there is only one element in your example it must have an index of zero. Accordingly you need
my $JobId = "$next->{'Job.0.Id'}";
I'm starting with mongodb and I'm finding many difficulties with the following scheme.
{
"_id" : "AAA",
"events" : [
{
"event" : "001",
"time" : 1456823333
},
{
"event" : "002",
"time" : 1456828888
},
{
"event" : "003",
"time" : 1456825555
},...
]
}
I want to get the events sorted by date and apply limit and skip.
I'm using the following query:
$op = array(
array('$match' => array('_id' => $userId)),
array('$unwind' => '$events'),
array('$sort' => array('events.time' => -1)),
array('$group' => array('_id' => '$_id',
'events' => array('$push' => '$events')))
//,array('$project' => array('_id' => 1, 'events' => array('$events', 0, 3)))
//,array('$limit' => 4)
//,array('$skip' => 3)
);
$result= Mongo->aggregate('mycollection', $op);
I have tried everything to filter $project or $limit and $skip but none of it works.
How should I apply the limit and skyp conditions in events?
If I do not apply the conditions of "limit" above the result is ordered correctly.
Result:
{ "waitedMS":0,
"result":[
{
"_id":"AAA",
"events":[
{
"event":"002",
"time":1456828888,
},
{
"event":"003",
"time":1456825555,
},
{
"event":"001",
"time":1456823333,
},...
}
],
"ok":1
}
Order correctly but I can not limit the number of results for paging.
I have to check the two fields having same values in same table in laravel 5. I am using Mongodb.
{
"id": "565d23ef5c2a4c9454355679",
"title": "Event1",
"summary": "test",
"total": NumberInt(87),
"remaining": NumberInt(87),
"status": "1"
}
I need to check "total" and "remaining" fields are same. How to write query in laravel 5.1. Please help.
One approach you could take would be using the aggregation framework methods from the raw MongoDB collection object provided from the underlying driver. In the mongo shell, you would essentially run the following aggregation pipeline operation to compare the two fields and return the documents which satisfy that criteria:
db.collection.aggregate([
{
"$project": {
"isMatch": { "$eq" : ["$total", "$remaining"] }, // similar to "valueof(total) == valueof(remaining)"
"id" : 1,
"title" : 1,
"summary" : 1,
"total" : 1,
"remaining" : 1,
"status" : 1
}
},
{
"$match": { "isMatch": true } // filter to get documents that only satisfy "valueof(total) == valueof(remaining)"
}
]);
Or using the $where operator in the find() query:
db.collection.find({ "$where" : "this.total == this.remaining" })
Thus in laravel, you can get the documents using raw expressions as follows
$result = DB::collection("collectionName") -> raw(function ($collection)
{
return $collection->aggregate(array(
array(
"$project" => array(
"id" => 1,
"title" => 1,
"summary" => 1,
"total" => 1,
"remaining" => 1,
"status" => 1,
"isMatch" => array(
"$eq" => array( "$total", "$remaining" )
)
)
),
array(
"$match" => array(
"isMatch" => true
)
)
));
});
In the case of $where, you can inject the expressions directly into the query:
Model::whereRaw(array("$where" => "this.total == this.remaining"))->get();
Or using the raw expression on the internal MongoCollection object executed on the query builder. Note that using the raw() method requires using a cursor because it is a low-level call:
$result = Model::raw()->find(array("$where" => "this.total == this.remaining"));
Collectionname::whereRaw(array('$where' => "this.filed1 > this.field2"))
I'm very much a noob when it comes to MapReduce and I have been pulling my hair out with this issue. Hopefully someone can give me a hand.
My Goal: Get the product revenue and a count of the units sold.
Transactions Collection sample document where i'm querying from:
{ "_id" : ObjectId( "xxxxxxxxxx" ),
"MerchantID" : { "$ref" : "merchants",
"$id" : ObjectId( "xxxxxxxx" ) },
"TransactionSocialKey" : "xxxxxxxx",
"PurchaseComplete: true,
"Products" : [
{ "ProductID" : { "$ref" : "products",
"$id" : ObjectId( "4ecae2b9cf72ab1f6900xxx1" ) },
"ProductPrice" : 14.99,
"ProductQuantity" : "1" },
{ "ProductID" : { "$ref" : "products",
"$id" : ObjectId( "4ecae2b9cf72ab1f690xxx2" ) },
"ProductPrice" : 14.99,
"ProductQuantity" : "1" } ],
"DateTimeCreated" : Date( 1321919161000 ) }
As you can see I have an embedded array called Products with the ProductID, Product Price, and Product Quantity.
My Map Function
map = function(){
if(this.PurchaseComplete === true){
this.Products.forEach(function(Product){
if(Product.ProductID.$id.toString() == Product_ID.toString()){
emit(Product_ID, {
"ProductQuantity" : Product.ProductQuantity,
"ProductPrice" : Product.ProductPrice,
"ProductID" : Product.ProductID.$id.toString()
});
}
});
}
}
So with this i'm only going to emit Transactions that were completed. If the Transaction was completed I'm looping through the Products array and if the Product.ProductID.$id is equal to the Product_ID that I set in the MapReduce Scope then I'm going to emit the Product from the set.
For testing sake I've set up my Reduce function as:
reduce = function(key, Product_Transactions){
return {"Transactions" : Product_Transactions};
}
For some odd reason i'm getting this sort of result:
[results] => Array
(
[0] => Array
(
[_id] => MongoId Object
(
[$id] => 4ecae2b9cf72ab1f6900xxx1
)
[value] => Array
(
[Transactions] => Array
(
[0] => Array
(
[Transactions] => Array
(
[0] => Array
(
[ProductQuantity] => 1
[ProductPrice] => 14.99
[ProductID] => 4ecae2b9cf72ab1f6900xxx1
)
[1] => Array
(
[ProductQuantity] => 1
[ProductPrice] => 14.99
[ProductID] => 4ecae2b9cf72ab1f6900xxx1
)
It Continues…
)
)
[1] => Array
(
[ProductQuantity] => 1
[ProductPrice] => 12.74
[ProductID] => 4ecae2b9cf72ab1f6900xxx1
)
[2] => Array
(
[ProductQuantity] => 1
[ProductPrice] => 12.74
[ProductID] => 4ecae2b9cf72ab1f6900xxx1
)
)
)
)
)
I'm not sure why I'm getting this odd embedded array. The emit key is always the same and never changes. I'm really lost for ideas on where to start trouble shooting. Any help or guidance would be appreciated.
Output of map should be in the same format that reduce consumes and produces. The idea is that reduce may run in parallel and/or against partially reduced results.
Here's how your code should look like (pseudo-code)
var map = function() {
if(some condition) {
emit(product_id, {Transactions: [{ // <= note the array here!
"ProductQuantity" : Product.ProductQuantity,
"ProductPrice" : Product.ProductPrice,
"ProductID" : ID
}]})
}
};
var reduce = function(key, vals) {
var result = {Transactions: []};
vals.forEach(function(v) {
v.Transactions.forEach(t) {
result.Transactions.push(t);
}
});
return result;
}