I have about 60k documents in my database and I cannot query for them. The only way I have been able to successfully retrieve them is with the hydrate(false) option.
I have tried both:
$dm = $this->get('doctrine_mongodb')->getManager();
$qb = $dm->createQueryBuilder('BConwayWebsiteBundle:Business')
->eagerCursor(true);
$query = $qb->getQuery();
$results = $query->execute();
and
$dm = $this->get('doctrine_mongodb')->getManager();
$qb = $dm->createQueryBuilder('BConwayWebsiteBundle:Business');
$query = $qb->getQuery();
$results = $query->execute();
If I dump $results->isInitialized() it is false. I believe that is because I have not utilized it yet at that point. Any accesing I try to do causes the memory issue though. Even something as basic as $results->count().
[2013-10-01 23:54:55] doctrine.INFO: MongoDB query: {"find":true,"query":[],"fields":[],"db":"ClosedForTheHoliday","collection":"businesses"} [] []
[2013-10-01 23:54:55] doctrine.INFO: MongoDB query: {"limit":true,"limitNum":null,"query":[],"fields":[]} [] []
[2013-10-01 23:54:55] doctrine.INFO: MongoDB query: {"skip":true,"skipNum":null,"query":[],"fields":[]} [] []
[2013-10-01 23:54:55] doctrine.INFO: MongoDB query: {"sort":true,"sortFields":[],"query":[],"fields":[]} [] []
[2013-10-01 23:54:56] emergency.EMERGENCY: Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) {"type":1,"file":"/media/sf_sites/cfth_com/vendor/doctrine/mongodb/lib/Doctrine/MongoDB/Cursor.php","line":237} []
UPDATE: If I limit my results to 2000, it works. Furthermore, if I select only the two fields that I am going to use after the query, I can limit it to 15000 and have success. When I limit it to 2000 the dev debug bar says that the query used 120MB of memory. I need to be able to iterate through all of the results, and it cannot be paginated. It seems like that should be possible without having 100MB+ queries...
I only needed distinct data from two fields, so this is what I ended up doing:
$dm = $this->get('doctrine_mongodb')->getManager();
// Get organization name from all businesses (distinct)
$organizations = $dm->createQueryBuilder('BConwayWebsiteBundle:Business')
->distinct('organization')
->getQuery()
->toArray();
// Get business name from all businesses (distinct)
$names = $dm->createQueryBuilder('BConwayWebsiteBundle:Business')
->field('organization')->equals('')
->field('organization')->equals(null)
->distinct('name')
->getQuery()
->toArray();
// Create new array from results of both queries
$businesses = array_merge($organizations, $names);
// Filter out any null or empty values
$businesses = array_filter($businesses, function($item) {
return (!is_null($item) && strlen($item) > 0);
});
// Filter out any duplicates
array_unique($businesses);
// Sort array case-insensitive
sort($businesses, SORT_STRING | SORT_FLAG_CASE);
Since Doctrine MongoDB ODM is a persistence manager (likewise with Doctrine ORM 2.x for relational databases), references to hydrated objects are stored internally and will not be recovered by PHP's garbage collection as easily as if you were iterating over non-hydrated array results that quickly leave scope.
For any batch operations in Doctrine, you'll want to either periodically clear() the object manager of all managed objects, or manually detach() individual objects. A periodic clear() would probably be easiest in your case, as you could do it every X iterations through the results. You can find some code examples and a discussion of this solution in this blog post. Although it's written from the perspective of the ORM and its EntityManager class, ODM's DocumentManager implements the same ObjectManager interface from the Doctrine Common library, which is where you'll find the detach() and clear() methods.
Related
Environment : MongoDb 3.2, Morphia 1.1.0
So lets say i am having a collection of Employees and Employee entity has several fields. I need to do something like apply multiple filters (conditional) and return a batch of 10 records per request.
pesudocode as below.
#Entity("Employee")
Employee{
String firstname,
String lastName,
int salary,
int deptCode,
String nationality
}
and in my EmployeeFilterRequesti carry the request parameter to the dao
EmployeeFilterRequest{
int salaryLessThen
int deptCode,
String nationality..
}
Pseudoclass
class EmployeeDao{
public List<Employee> returnList;
public getFilteredResponse(EmployeeFilterRequest request){
DataStore ds = getTheDatastore();
Query<Employee> query = ds.createQuery(Emploee.class).disableValidation();
//conditional request #1
if(request.filterBySalary){
query.filter("salary >", request.salary);
}
//conditional request #2
if(request.filterBydeptCode){
query.filter("deptCode ==", request.deptCode);
}
//conditional request #3
if(request.filterByNationality){
query.filter("nationality ==", request.nationality);
}
returnList = query.batchSize(10).asList();
/******* **THIS IS RETURNING ME ALL THE RECORDS IN THE COLLECTION, EXPECTED ONLY 10** *****/
}
}
SO as explained above in the code.. i want to perform conditional filtering on multiple fields. and even if batchSize is present as 10, i am getting complete records in the collection.
how to resolve this ???
Regards
Punith
Blakes is right. You want to use limit() rather than batchSize(). The batch size only affects how many documents each trip to the server comes back with. This can be useful when pulling over a lot of really large documents but it doesn't affect the total number of documents fetched by the query.
As a side note, you should be careful using asList() as it will create objects out of every document returned by the query and could exhaust your VM's heap. Using fetch() will let you incrementally hydrate documents as you need each one. You might actually need them all as a List and with a size of 10 this is probably fine. It's just something to keep in mind as you work with other queries.
I'm working on super simple search across multiple fields in a document to see if any of them has a single value. (Note: some fields are using regex to search if value is contained in string). Using query builder I constructed the following.
public function search($value, $limit, $offset=0, $orderby = '', $order='' )
{
$regexVal = new \MongoRegex('/^.*(\b'.str_replace(' ', '\s', $value).'\b).*?$/i');
$query = $this->repository->createQueryBuilder();
$query->addOr($query->expr()->field('location')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.first_name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.last_name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.email')->equals($value));
$query->addOr($query->expr()->field('email')->equals($value));
$query->addOr($query->expr()->field('organization')->equals($value));
$query->limit($limit)
->skip($offset);
if( ! empty($orderby) && $order ){
$query->sort($orderby, $order);
}
return $query->getQuery()->execute();
}
If I dump out the constructed query values I get the following array in this gist. https://gist.github.com/jchamb/04a0400c989cd28b1841 The extra association field in there is being added by a Doctrine Filter.
Through Query builder I don't get any results, however if I construct the query myself and run it in an admin app like genghis, I get the expected single document result.
Actual written mongodb string looks like this. https://gist.github.com/jchamb/ce60829480576a88290d
This project is a zend2 app that was already using doctrine and mongo. I'm not much of an expert with mongo in general so I'm not sure what i'm doing wrong inside of Query Builder that i'm not getting the same result as executing the query directly. I can't find any info on stack or the query builder docs that gives any extra clues for the multiple addOrs syntax either.
Any help or direction would be really appreciated, in the most basic form I need query builder to get a document where association = x and ( field1 = val or field2 = value).
Thanks!
Really unsure what the exact issue was with the above, but after playing around, switching the order of query builder around fixes the problem.
public function search($value, $limit, $offset=0, $orderby = '', $order='' )
{
$regexVal = new \MongoRegex('/^.*(\b'.str_replace(' ', '\s', $value).'\b).*?$/i');
$query = $this->repository->createQueryBuilder()
->find()
->limit($limit)
->skip($offset);
$query->addOr($query->expr()->field('location')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.first_name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.last_name')->equals($regexVal));
$query->addOr($query->expr()->field('mappedData.email')->equals($value));
$query->addOr($query->expr()->field('email')->equals($value));
$query->addOr($query->expr()->field('organization')->equals($value));
if( ! empty($orderby) && $order ){
$query->sort($orderby, $order);
}
return $query->getQuery()->execute();
}
Would love to still hear some feedback about why this works and the above didn't if anyone know more about the internals of query builder.
this is becoming frustrating beyond imagination.
I need to get the column names from a table using Eloquent ORM in Laravel 5 combined with MongoDB. I have found some examples, but none of them is working for me as they are probably made for SQL specifically. I tried this and this without success, any idea?
Thanks!
It would be best to use the raw() method in this case and use the native MongoCollection methods like find() to iterate over the collection and get the keys in the documents array:
// Returns an array of field names from a collection of User models.
$keys = DB::collection('users')->raw(function($collection)
{
$cursor = $collection->find();
$array = iterator_to_array($cursor);
$fields = array();
foreach ($array as $k=>$v) {
foreach ($v as $a=>$b) {
$fields[] = $a;
}
}
return array_values(array_unique($fields));
});
MongoDB doesn't have columns or tables - the whole point is that it's schemaless therefore there are no columns for you to get the names of.
Because every document can be different, you'd need to get all documents in your collection and build an array of unique keys that each document contains.
See this answer:
MongoDB Get names of all keys in collection
Hello ive got the following code:
$primer = function($dm, $className, $fieldName, $ids, $hints) {
$repository = $dm->getRepository($className);
$qb = $repository->createQueryBuilder()
->field('id')->in($ids)
->field('images')->prime(true);
$query = $qb->getQuery();
$query->execute()->toArray();
};
$qb = $followRepo
->createQueryBuilder()
->field('isActive')->equals(true)
->field('target')->prime($primer)
->field('follower')->references($return['user'])
->field('target.$ref')->equals('boards')
->field('target.createdBy.type')->equals('user') // here i dont know how to handle this
->sort('created', 'desc')
Is it even possible in mongo to query via target.createdBy.type?
target.createdBy is also ref.
Yes, it is possible to query on sub document properties using the dot notation, as per
the official documentation. I am pretty sure it wont work with a referencing in 1 step though.
I have not clearly issue during query using two criterials like Id and Other. I use a Repository storing some data like id,iso,value. I have created an index("_id","Iso") to performs queries but queries are only returning my cursor if i use only one criterial like _id, but is returning nothing if a use two (_id, Iso) (commented code).
Are the index affecting the response or the query method are failing?
use :v1.6.5 and C# official.
Sample.
//Getting Data
public List<BsonObject> Get_object(string ID, string Iso)
{
using (var helper = BsonHelper.Create())
{
//helper.Db.Repository.EnsureIndex("_Id","Iso");
var query = Query.EQ("_Id", ID);
//if (!String.IsNullOrEmpty(Iso))
// query = Query.And(query, Query.EQ("Iso", Iso));
var cursor = helper.Db.Repository.FindAs<BsonObject>(query);
return cursor.ToList();
}
}
Data:
{
"_id": "2345019",
"Iso": "UK",
"Data": "Some data"
}
After that I have Updated my data using Update.Set() methods. I can see the changed data using MongoView. The new data are correct but the query is always returning the sames olds values. To see these values i use a page that can eventually cached, but if add a timestamp at end are not changing anything, page is always returning the same olds data. Your comments are welcome, thanks.
I do not recall offhand how the C# driver creates indexes, but the shell command for creating an index is like this:
db.things.ensureIndex({j:1});
Notice the '1' which is like saying 'true'.
In your code, you have:
helper.Db.Repository.EnsureIndex("_Id","Iso");
Perhaps it should be:
helper.Db.Repository.EnsureIndex("_Id", 1);
helper.Db.Repository.EnsureIndex("Iso", 1);
It could also be related to the fact that you are creating indexes on "_Id" and the actual id field is called "_id" ... MongoDB is case sensitive.
Have a quick look through the index documentation: http://www.mongodb.org/display/DOCS/Indexes