When I am working on above 5,000,000 records in mongodb, it shows this error "Read timed out after reading 0 bytes, waited for 30.000000 seconds" in find() query. Please any one help me.
In PHP you can set timeout(-1) to your cursor.
PHP Example:
$conn = new MongoClient("mongodb://primary:27017,secondary:27017", array("replicaSet" => "myReplSetName"));
$db = $conn->selectDB(DBname);
$collection = $db->selectCollection(collection);
$cursor = $collection->find();
$cursor->timeout(-1);
Node.js Example:
// Database connect options
var options = { replset: { socketOptions: { connectTimeoutMS : conf.dbTimeout }}};
// Establish Database connection
var conn = mongoose.connect(conf.dbURL, options);
In PHP you could add the parameter socketTimeoutMS to the other parameters of connection string, in this case to 90 seconds.
$conn = new MongoClient("mongodb://primary:27017,secondary:27017", array("socketTimeoutMS" => "90000"));
Greetings!
Take a look at the mongodb log file and find your query in it -- how long does it take to execute? If it does take a long time, have you added indexes? Are they being used? Cut/paste the query from mongodb log file and try it from mongo shell -- and add ".explain()" at the end. It will tell you the execution plan that MongoDB is performing -- and perhaps you can attack your problem from that side. If your queries really do take longer than 30 seconds, you most likely need to address it anyway -- regardless of the driver timeout issues.
ensureIndex keys and try
MongoCursor::$timeout = 600000;
I've spotted this problem on removing 1-2kk logs records using php driver.
Basically i had to add timeout (i'm using indexes, just db is a bit huge)
$dtObject = new DateTime();
$dtObject->modify("- " . $interval);
$date = new MongoDate($dtObject->format("U"));
$filter = array('dtCreated' => array('$lt' => $date));
$num = $this->collection->count($filter);
if ($num > 0)
$this->collection->remove($filter)->timeout(-1);
return $num;
This worked for me
Related
Is there a way to check if pg_prepare already executed and remove it from the session?
Seems like pg_close doesn't remove prepared statement from the session. Kind of seems like a bug in php, but maybe I'm missing something or maybe there is a workaround.
public static function readSubdomains($dcName, $filter = null) {
// ...
$conn = pg_pconnect($connectionString);
// ...
$result = pg_prepare($conn, "subdomains", "SELECT subdomain
from tenants
where $where
order by 1 asc
");
$result = pg_execute($conn, "subdomains", $params);
// ...
pg_close($conn);
}
Second call to readSubdomains shows a warning like this:
Warning: pg_prepare(): Query failed: ERROR: prepared expression "subdomains" already exists in inc/TestHelper.php on line 121
Always check the official manuals for this sort of stuff.
https://www.postgresql.org/docs/current/view-pg-prepared-statements.html
Oh - if pg_close isn't dropping prepared statements then it isn't closing the connection. You might have some connection pooling involved.
I am trying to select rows which are older than 7 days from current date. Database used is DB2 version 9.
Can you please tell me how exactly can I use the datetime in the query? The date table field is of type timestamp.
I am able to manually run the query without issues. However, when I am using in the prepared statement,
The app hangs when executing the query result = pselect.executeQuery(); as a result of which we need to restart db2 instance in order to clear it.
Can you please help what might be the issue? I do not see any exceptions at all. Other parts of the code works fine if I remove the select_query part.
try{
String select_query = "SELECT URL_ID ,URLVAL FROM URL_TAB WHERE " +
"UPDATED_DATE < TIMESTAMP(CURRENT_DATE - 7 DAYS, '00.00.00')";
System.out.println("select_query=" + select_query);
conn = JDBCDataObjectFactoryManager
.getConnection("JDBCConnectionFactory-SDE");
pselect = conn.prepareStatement(select_query);
System.out.println("pselect=" + pselect);
try{
System.out.println("inside try");
result = pselect.executeQuery();
System.out.println("result=" + result);
}catch(Exception e){
System.out.println("inside catch");
System.out.println("error message==============>"+e.getMessage());
}
if ((result != null) && (result.next())) {
System.out.println("3 >>>>>>>>>>>>>>>>>>>>>>>>>");
url_id = result.getInt(1);
url = result.getString(2);
}//end if
There are two possibilities: either the query is in a lock wait, or it runs for so long that it appears to be hung.
Check what is the value of LOCKWAIT database configuration parameter --by default it is -1, which means infinity, and you normally want to set it to a more reasonable value, typically 30 or 60 seconds. If it is the lock wait that causes your application to "hang", you would get an exception instead, which will help you to debug further.
If the issue is caused by the poor query performance, you'll need to work with your DBAs to figure out the root cause and resolve it.
In my symfony2 command, I am running a script that inserts hundreds of thousands of urls (as string) into a document.
Here are the basic structures of the 2 documents I'm using. Before the program is run, there are thousands of ParentDocuments already inside the mongodb, but zero ChildDocuments:
ParentDocument:
$id:id
$subDocument:OneToManyReference(ChildDocument)
$etc:everythingelse
ChildDocument:
$id:id
$url:string
$parentDocument:ManyToOneReference(ParentDocument)
And my Command code:
$dm = $this->getContainer()->get('doctrine_mongodb.odm.document_manager');
$parentDocuments = $dm->repository('My:Bundle:ParentDocument')->findAll();
while ($parentDocument = $parentDocuments->getNext()) {
//Returns an array of hundreds of thousands urls
$urls = $this->somehowFetchUrlsRelatedToTheParentDocument($parentDocument);
foreach ($urls as $url) {
$subDocument = new SubDocument();
$subDocument->setUrl($url);
$subDocument->setParentDocument($parentDocument);
$dm->persist($subDocument);
}
$dm->flush();
}
When I run this simple command, the write speed at first is incredibly fast. However, in the case of inserting millions of rows, the write speeds become significantly slower. As slow as 1 write per second after the command has been running for 10 minutes, making the code extremely ineffective.
My first attempt at fixing this problem was to clear the document manager right after it flushes using $dm->clear();
But this meant that the document manager would lose track of the current ParentDocument. So my solution was this:
$dm = $this->getContainer()->get('doctrine_mongodb.odm.document_manager');
$parentDocumentCursors = $dm->repository('My:Bundle:ParentDocument')->findAll();
$parentDocuments = array();
while ($parentDocument = $parentDocumentCursors->getNext()) {
array_push($parentDocuments, $parentDocument);
}
$dm->clear();
unset($dm);
$dm = $this->getContainer()->get('doctrine_mongodb.odm.document_manager');
foreach ($parentDocuments as $parentDocument) {
$urls = $this->somehowFetchUrlsRelatedToTheParentDocument($parentDocument);
foreach ($urls as $url) {
$subDocument = new SubDocument();
$subDocument->setUrl($url);
$subDocument->setParentDocument($parentDocument);
$dm->persist($subDocument);
}
$dm->flush();
$dm->clear();
}
This solved the problem. The write speeds were consistently fast throughout the whole execution of the program and millions of rows were able to be inserted without gradual delay.
However, this feels like a bad practice and a quick fix hack. What is the best practice for inserting millions of rows in Symfony2 using document manager without read/write speeds becoming slow?
I would avoid using Symfony's document manager and use the batchInsert() function directly. This is described in the documentation at http://php.net/manual/en/mongocollection.batchinsert.php It feels to me like Doctrine's ODM is actually hurting you here.
In order to do a bulk insert in doctrine you would need to move your flush outside of your loop. Consider the scenario below where you would persist in the foreach then flush when the foreach is completed. Your only catch will be that you will not be able to query any of the data being inserted in the batch until after the flush.
$dm = $this->getContainer()->get('doctrine_mongodb.odm.document_manager');
foreach ($parentDocuments as $parentDocument) {
$urls = $this->somehowFetchUrlsRelatedToTheParentDocument($parentDocument);
foreach ($urls as $url) {
$subDocument = new SubDocument();
$subDocument->setUrl($url);
$subDocument->setParentDocument($parentDocument);
$dm->persist($subDocument);
}
}
$dm->flush();
$dm->clear();
Another option is to do a a push,pushall, or addto set.
One issue to consider is you will need to use stdClass in php in order to add an object.
I find this to be the quickest way to update a subdocument.
For example:
$dm->createQueryBuilder('My:Bundle:ParentDocument')
->update()
->field('subDocument')->push( (object) array('url'=> $url) )
->field('id')->equals( $parentDocumentId )
->getQuery()
->execute();
I am working with the neo4j-rest-graphdb an just tried to use Cypher for fetching a simple Node result.
CypherParser parser = new CypherParser();
ExecutionEngine engine = new ExecutionEngine(graphDbService);
Query query = parser.parse( "START referenceNode = node (0) " +
"MATCH referenceNode-[PRODUCTS_REFERENCE]->products-[PRODUCT]->product " +
"RETURN product.productName " +
"ORDER BY product.productId " +
"SKIP 20"
"LIMIT 10");
ExecutionResult result = engine.execute( query );
Iterator<Map<String, Object>> iterator = result.javaIterator();
What is the best practise to iterate through the result? The last line causes my service to hang for ~6 sec. Without the iterator at the end the application is quiet fast. I also tried the webadmin cypher terminal, the results are fetched within 50ms. Am i doing something wrong?
In your case all the cypher operations (graph-matching, filtering etc. would go over the wire which is awfully chatty and slow) you don't want that !
The neo4j-rest-graphdb supports remote execution of cypher out of the box:
Just do, something like shown in this testcase:
RestCypherQueryEngine queryEngine = new RestCypherQueryEngine(restGraphDatabase.getRestAPI());
final String queryString = "start n=node({reference}) return n";
Map params = MapUtil.map("reference",0);
final Node result = (Node) queryEngine.query(queryString, params).to(Node.class).single();
assertEquals(restGraphDatabase.getReferenceNode(), result);
If I understood you correctly, graphDbService is a REST graph database, correct?
If you want to use Cypher on the Server, you should instead use the CypherPlugin. Look here: http://docs.neo4j.org/chunked/snapshot/cypher-plugin.html
I hope this helps,
Andrés
memchached is useful for caching and looking up of single independent records. For the multiple records returned from doing a SELECT operation, how could I make good use of the memcached for caching and looking up later?
I didn't get you. If you use php or .net (Enyim client) you can store your result in to an object and set it in to the memcached. (Client will serialize the object and store it in the memcached.)
Following example will store one or more records (results) returned by the db query.
//Init Memcache, Try avoid multiple init if possible because this is a costly operation
$memcache = new Memcache;
$memcache->connect('localhost', 11211) or die ("Could not connect");
mysql_pconnect("localhost","root","");
mysql_select_db("YOURDB");
$query = "SELECT * FROM table where `firstname`='dasun';";
$key = md5($query);
$get_result = $memcache->get($key);
if ($get_result) {
print_r($get_result);
echo "It's a hit! :)";
}
else {
$result = mysql_query($query);
$row = mysql_fetch_array($result);
print_r($row);
// Store the result for 5 minutes
$memcache->set($key, $row, TRUE, 300);
echo "Not a hit. :(";
}