mongodb looping collection + save, objects returned several times - mongodb

I'm writing a pretty big migration and had this code (coffeescript):
db.users.find().forEach (user)->
try
#some code changing the user depending on the old state
db.users.save(user)
print "user_ok: #{user._id}"
catch error
print "user_error: #{user._id}, error was: #{error}"
Some errors occured. But they occured on already processed users:
user_ok: user_1234
#many logs
user_error: user_1234 ...
How come the loop takes already processed objects?
I ended up doing:
backup = { users: [] }
db.users.find().forEach (user)->
try
#some code changing the user depending on the old state
backup.users.push user
print "user_ok: #{user._id}"
catch error
print "user_error: #{user._id}, error was #{error}"
#loop backup and save
And it works nice now, but it seems really weird. What's the point behind all that please?

When you modify an object, it might be moved by the database. The database needs to take additional care to remember which objects have been visited already. This feature is called snapshotting, you can ask for a snapshotted query using
db.collection.find().snapshot()
However, even this doesn't make guarantees about objects that were inserted or deleted during the cursor iteration. A few more caveats are explained in the link to the documentation.
Another option is to perform an $orderby on an invariable unique index. Ideally, that index is also monotonic, so if you are using ObjectIds as primary keys then the _id field comes in pretty handy, like
db.collection.find().sort({"_id" :1});

Related

How does resource.data.size() work in firestore rules (what is being counted)?

TLDR: What is request.resource.data.size() counting in the firestore rules when writing, say, some booleans and a nested Object to a document? Not sure what the docs mean by "entries in the map" (https://firebase.google.com/docs/reference/rules/rules.firestore.Resource#data, https://firebase.google.com/docs/reference/rules/rules.Map) and my assumptions appear to be wrong when testing in the rules simulator (similar problem with request.resource.data.keys().size()).
Longer version: Running into a problem in Firestore rules where not being able to update data as expected (despite similar tests working in the rules simulator). Have narrowed down the problem to point where can see that it is a rule checking for request.resource.data.size() equaling a certain number.
An example of the data being passed to the firestore update function looks like
Object {
"parentObj": Object {
"nestedObj": Object {
"key1": Timestamp {
"nanoseconds": 998000000,
"seconds": 1536498767,
},
},
},
"otherKey": true,
}
where the timestamp is generated via firebase.firestore.Timestamp.now().
This appears to work fine in the rules simulator, but not for the actual data when doing
let obj = {}
obj.otherKey = true
// since want to set object key name dynamically as nestedObj value,
// see https://stackoverflow.com/a/47296152/8236733
obj.parentObj = {} // needed for adding nested dynamic keys
obj.parentObj[nestedObj] = {
key1: fb.firestore.Timestamp.now()
}
firebase.firestore.collection('mycollection')
.doc('mydoc')
.update(obj)
Among some other rules, I use the rule request.resource.data.size() == 2 and this appears to be the rules that causes a permission denied error (since commenting out this rules get things working again). Would think that since the object is being passed with 2 (top-level) keys, then request.resource.data.size()=2, but this is apparently not the case (nor is it the number of keys total in the passed object) (similar problem with request.resource.data.keys().size()). So there's a long example to a short question. Would be very helpful if someone could clarify for me what is going wrong here.
From my last communications with firebase support around a month ago - there were issues with request.resource.data.size() and timestamp based security rules for queries.
I was also told that request.resource.data.size() is the size of the document AFTER a successful write. So if you're writing 2 additional keys to a document with 4 keys, that value you should be checking against is 6, not 2.
Having said all that - I am still having problems with request.resource.data.size() and any alternatives such as request.resource.size() which seems to be used in this documentation
https://firebase.google.com/docs/firestore/solutions/role-based-access
I also have some places in my security rules where it seems to work. I personally don't know why that is though.
Been struggling with that for a few hours and I see now that the doc on Firebase is clear: "the request.resource variable contains the future state of the document". So with ALL the fields, not only the ones being sent.
https://firebase.google.com/docs/firestore/security/rules-conditions#data_validation.
But there is actually another way to ONLY count the number of fields being sent with request.writeFields.size(). The property writeFields is a table with all the incoming fields.
Beware: writeFields is deprecated and may stop working anytime, but I have not found any replacement.
EDIT: writeFields apparently does not work in the simulator anymore...

How do I structure my Geofire posts in my firebase database

I have noticed on all other posts that any node/key with children was saved in the database in quotes and mine are not.
mine
Locations{
indexOn: "g"
}
others/what I assume it should be
Locations{
".indexOn": "g"
}
I didn't think anything of it till I came up on a similar error as this with the ".indexOn". I tried adding ".indexOn" but i received the error that no key can have the symbol .(along with a few other characters that aren't allowed) so I put it in without the . like so:
Locations{
(specific id){
g: "345jh3i5jh"
l{
0: 37
1: -120
}
indexOn: "g"
user: "0987435098723098Gjhf90"
}
}
So It seems to work as the observeEventType is returning the correct result sometimes but it still also gives me the error that i need to put ".indexOn" in database. How do I do this?
Any help and explicit examples would be appreciated as there are probably more problems with everything than I addressed. Doing this in swift and thanks for the read!
You seem to be trying to add index definitions to your database. That is indeed a great way to ensure the server can order and filter the data before returning the results to your app.
Indexes are defined by adding them to your Firebase Database rules in the Firebase Console > Database > Rules. Don't try to add them to your actual database, since:
you won't be able to save them, since they have a . in their key, which is an illegal character in the database
adding them won't help, since the database server only searches for .indexOn definitions in the database rules
See the Firebase documentation on adding indexes for more.

Cannot update one field at a time with VSTO for Word

When fields are nested, there is a problem.
foreach (Word.Field field in this.Application.ActiveDocument.Fields)
{
field.Update();
text = field.Result.Text;
}
The above code does not work.
The process starts, but winds up in an endless loop or some other process that hangs the system.
Thinking about it, I can surmise that when you update a field, it might have an effect on the fields collection - thus, the loop fails.
Does anyone have any ideas on implementing this?
P.S. I know there is a Document.UpdateFields() method to update ALL fields. However, there are reasons why I cannot use this and need to only update specific field types.
My apologies! I was going to give an example of a nested field but was trying to test some more before sending anyone (Jack) on a goose-chase.
I waited and waited and waited, and after a good 2 or 3 minutes, it finished. After the last field, it crashed with this message:
Object has been deleted.
The error was generated from the following line inside the loop:
string text = field.Code.Text;
The template is being tested on mergefields that are not being found because I am testing without database connectivity. It would be odd, but explainable, that it goes through all the fields and then, at the end of the day, the very OUTER IF field's result is "Error! Reference source not found." But I still don't get why this could happen.
Nor do I understand why looping takes 3 minutes while a call to document.Fields.Update() will do the same thing in about 1 second and NOT result in the error described above.
Again, my apologies. I never considered updating inside a loop would be vastly slower that a call to doc.fields.update().

Manipulating form input values after submission causes multiple instances

I'm building a form with Yii that updates two models at once.
The form takes the inputs for each model as $modelA and $modelB and then handles them separately as described here http://www.yiiframework.com/wiki/19/how-to-use-a-single-form-to-collect-data-for-two-or-more-models/
This is all good. The difference I have to the example is that $modelA (documents) has to be saved and its ID retrieved and then $modelB has to be saved including the ID from $model A as they are related.
There's an additional twist that $modelB has a file which needs to be saved.
My action code is as follows:
if(isset($_POST['Documents'], $_POST['DocumentVersions']))
{
$modelA->attributes=$_POST['Documents'];
$modelB->attributes=$_POST['DocumentVersions'];
$valid=$modelA->validate();
$valid=$modelB->validate() && $valid;
if($valid)
{
$modelA->save(false); // don't validate as we validated above.
$newdoc = $modelA->primaryKey; // get the ID of the document just created
$modelB->document_id = $newdoc; // set the Document_id of the DocumentVersions to be $newdoc
// todo: set the filename to some long hash
$modelB->file=CUploadedFile::getInstance($modelB,'file');
// finish set filename
$modelB->save(false);
if($modelB->save()) {
$modelB->file->saveAs(Yii::getPathOfAlias('webroot').'/uploads/'.$modelB->file);
}
$this->redirect(array('projects/myprojects','id'=>$_POST['project_id']));
}
}
ELSE {
$this->render('create',array(
'modelA'=>$modelA,
'modelB'=>$modelB,
'parent'=>$id,
'userid'=>$userid,
'categories'=>$categoriesList
));
}
You can see that I push the new values for 'file' and 'document_id' into $modelB. What this all works no problem, but... each time I push one of these values into $modelB I seem to get an new instance of $modelA. So the net result, I get 3 new documents, and 1 new version. The new version is all linked up correctly, but the other two documents are just straight duplicates.
I've tested removing the $modelB update steps, and sure enough, for each one removed a copy of $modelA is removed (or at least the resulting database entry).
I've no idea how to prevent this.
UPDATE....
As I put in a comment below, further testing shows the number of instances of $modelA depends on how many times the form has been submitted. Even if other pages/views are accessed in the meantime, if the form is resubmitted within a short period of time, each time I get an extra entry in the database. If this was due to some form of persistence, then I'd expect to get an extra copy of the PREVIOUS model, not multiples of the current one. So I suspect something in the way its saving, like there is some counter that's incrementing, but I've no idea where to look for this, or how to zero it each time.
Some help would be much appreciated.
thanks
JMB
OK, I had Ajax validation set to true. This was calling the create action and inserting entries. I don't fully get this, or how I could use ajax validation if I really wanted to without this effect, but... at least the two model insert with relationship works.
Thanks for the comments.
cheers
JMB

After querying DB I can't print data as well as text anymore to browser

I'm in a web scripting class, and honestly and unfortunately, it has come second to my networking and design and analysis classes. Because of this I find I encounter problems that may be mundane but can't find the solution to it easily.
I am writing a CGI form that is supposed to work with a MySQL DB. I can insert and delete into the DB just fine. My problem comes when querying the DB.
My code compiles fine and I don't get errors when trying to "display" the info in the DB through the browser but the data and text doesn't in fact display. The code in question is here:
print br, 'test';
my $dbh = DBI->connect("DBI:mysql:austinc4", "*******", "*******", {RaiseError => 1} );
my $usersstatement = "select * from users";
my $projstatment = "select * from projects";
# Get the handle
my $userinfo = $dbh->query($usersstatement);
my $projinfo = $dbh->query($projstatement);
# Fetch rows
while (#userrow = $userinfo->fetchrow()) {
print $userrow[0], br;
}
print 'end';
This code is in an if statement that is surrounded by the print header, start_html, form, /form, end_html. I was just trying to debug and find out what was happening and printed the statements test and end. It prints out test but doesn't print out end. It also doesn't print out the data in my DB, which happens to come before I print out end.
What I believe I am doing is:
Connecting to my DB
Forming a string the contains the command/request to the DB
Getting a handle for my query I perform on the DB
Fetching a row from my handle
Printing the first field in the row I fetched from my table
But I don't see why my data wouldn't print out as well as the end text. I looked in DB and it does in fact contain data in the DB and the table that I am trying to get data from.
This one has got me stumped, so I appreciate any help. Thanks again. =)
Solution:
I was using a that wasn't supported by the modules I was including. This leads me to another question. How can I detect errors like this? My program does in fact compile correctly and the webpage doesn't "break". Aside from me double checking that all the methods I do use are valid, do I just see something like text not being displayed and assume that an error like this occurred?
Upon reading the comments, the reason your program is broken is because query() does not execute an SQL query. Therefore you are probably calling an undefined subroutine unless this is a wrapper you have defined elsewhere.
Here is my original posting of helpful hints, which still apply:
I hope you have use CGI, use DBI, etc... and use CGI::Carp and use strict;
Look in /var/log/apache2/access.log or error.log for the bugs
Realize that the first thing a CGI script prints MUST be a valid header or the web server and browser become unhappy and often nothing else displays.
Because of #3 print the header first BEFORE you do anything, especially before you connect to the database where the script may die or print something else because otherwise the errors or other messages will be emitted before the header.
If you still don't see an error go back to #2.
CGIs that use CGI.pm can be run from a command line in a terminal session without going through the webserver. This is also a good way to debug.