I'm using mongodb's bulk write api to update/upsert several documents at once in an unordered fashion. I use the response to monitor number of upserts, updates, deletes, etc. Once in a while my code throws the CodecConfigurationException because it can't update fields such as BigInteger.
When such an exception is thrown do other commands go through successfully? From what I can tell, it does seem so
If other updates are being applied successfully then shouldn't such an error be part of the MongoBulkWriteException that allows for successful writes to be monitored and pin-points the unsuccessful writes along with the reason for write failure. How can one determine the failed update if we're updating, say 50000 documents at a time?
If no updates go through then is it that this exception is triggered even before the command is executed on the server side? Is this is a purely client side exception?
Related
I am a new for Postgres. Currently I hit a situation, I need to retrieve a record from DB/table, then update this record with some changes, then retrieve the updated record again to return to customer, these 3 operations are executed sequentially.
After I run the above 3 steps, sometimes it seems the record has not been updated. But in fact the record has been updated, I suspect when I retrieve the record, Postgres return a cached data rather than fresh data. Actually all DB operations are correct, I guess just Postgres returns cached records.
I am wondering if there is any mechanism to flush my updated data immediately?
Another question is, I am wondering which one is a good practice:
Update a record (actually write to DB), then immediately retrieve the record from DB, both operations are using statement to operate.
Update a record (actually write to DB), then don't retrieve record from DB, because we know updated data, we just use these data to return to customer. However, the record might fail to write DB.
Any ideas for the above?
Some programming languages do db calls asynchronously, meaning that your code is moving on to the next db operation without waiting for the first to finish. So, it could be as simple as using your language's "await" keyword to make sure you are waiting for your db to finish the "update record" before trying to read it again.
If you are writing raw sql or you know it is not an issue with making several db call asynchronously, you could try writing your update and read calls as a single transaction.
See https://www.postgresql.org/docs/14/tutorial-transactions.html if you're unfamiliar with writing transactions.
Suppose I do the following list of operations in MongoDB
Start a session
Start a transaction for that session
run an insert command with a new document
run a find command on the collection of the inserted document
Commit the transaction
End the session
I understand that outside of the transaction the insert done in the third step will not be visible until the transaction is committed, but what about within the transaction, will the find run in the fourth step see this new document or will it not?
Yes, a transactional find sees a document inserted in a previous transactional insert. You could assume the Read your own writes property.
Every time a transaction is started a new snapshot is created. Outside the transaction, the snapshot is obviously invisible: this is accomplished by using (and abusing if your transaction involves many updates) the WiredTiger cache. This cache is structured as a tree, similar to the following one1:
where each transactional operation is represented as a new update block that could in turn be chained to another update block.
Outside operations only see non-transactional tree entries, while transactional operations see all the entries added before the snapshot is taken + the update blocks for the given transaction.
I am aware that it is a very brief explanation on how MongoDB manages transaction atomicity, but if you are interested in understanding more on this, then I suggest you to read a report I have written. in the same repository you can find some scenarios for the most typical doubts.
1: image taken from Aly Cabral's presentation about How and when to use Multi-document transactions
I have a node.js web application connecting to MongoDB for which I want to profile DB performance. Each request received by the application is assigned a request-id and can cause multiple queries to be sent to MongoDB. I want to see this request-id value in each log line in MongoDB. Is there a way to do this? I would like to avoid adding always-true fields to each query like "req<id>": null because I suspect this may affect performance.
The docs reference a feature similar to this called Client Data at https://docs.mongodb.com/manual/reference/log-messages/ however this appears to be sent once per connection, and I'm looking for client data that changes multiple times even on the same single connection.
Try using cursor.comment() to record the request ID value into the log messages.
I am new in mongodb and i want to know How mongodb handels users requests.
What happened if the multiple users fire the multiple insert commands or read commands at the same time.
2:-When or where Snapshot coming in to the picture.(Which phase).
Multiple Inserts and Multiple Reads
MongoDB allows multiple clients to read and write the same data.
In order to ensure consistency, it uses locking and other concurrency control measures to prevent multiple clients from modifying the same piece of data simultaneously
Read this documentation it will give you complete info about concurrency
concurrency reference
MongoDB allows very fast writes and updates by default. The tradeoff is that you are not explicitly notified of failures.By default most drivers do asynchronous, ‘unsafe’ writes - this means that the driver does not return an error directly, similar to INSERT DELAYED with MySQL. If you want to know if something succeeded, you have to manually check for errors using getLastError.
MongoDB doesn't offer durability if you use the default configuration. It writes once every minute data to the disk.
This can be configured using j Option and Write Concern on the insert query.
write-concern reference
Snapshot
The $snapshot operator prevents the cursor from returning a document more than once because an intervening write operation results in a move of the document.
Even in snapshot mode, objects inserted or deleted during the lifetime of the cursor may or may not be returned.
snapshot reference
References: here and here
Hope it Helps!!
I am asking that question in the context of journaling in mongodb. As per the mongodb documentation. A write operation first come into the private view.So the Quetion is if multiple write operation have been performed at the same time,than multiple private view will be created...
2;-Checkpoints and snapshot:in the journaling process which point of place snapshot of data is available..?
I'm trying to debug an application (under PostgreSQL) and came across the following error:
"current transaction is aborted, commands ignored".
As far as I can understand a "transaction" is just a notion related to the underlying database connection.
If the connection has an auto commit "false", you can execute queries through the same Statement as long as it isn't failing. In which case you should rollback.
If auto commit is "true" then it doesn't matter as long as all your queries are considered atomic.
Using auto commit false, I get the aforementioned error by PostgreSQL even when a simple
select * from foo
fails, which makes me ask, under which SQLException(s) is a "transaction" considered invalid and should be rolled backed or not used for another query?
using MacOS 10.5, Java 1.5.0_16, PostgreSQL 8.3 with JDBC driver 8.1-407.jdbc3
That error means that one of the queries sent in a transaction has failed, so the rest of the queries are ignored until the end of the current transaction (which will automatically be a rollback). To PostgreSQL the transaction has failed, and it will be rolled back in any case after the error with one exception. You have to take appropriate measures, one of
discard the statement and start anew.
use SAVEPOINTs in the transaction to be able to get back to that point in time and try another path. (This is the exception)
Enable query logging to see which query is the failing one and why.
In any case the exact answer to your question is that any SQLException should mean a rollback happened when the end of transaction command is sent, that is when a COMMIT or ROLLBACK (or END) is issued. This is how it works, if you use savepoints you'll still be bound by the same rules, you'll just be able to get back to where you saved and try something else.
It seems to be a characteristic behaviour of PostgreSQL that is not shared by most other DBMS. In general (outside of PostgreSQL), you can have one operation fail because of an error and then, in the same transaction, can try alternative actions that will succeed, compensating for the error. One example: consider a merging (insert/update) operation. If you try to INSERT the new record but find that it already exists, you can switch to an UPDATE operation that changes the existing record instead. This works fine in all the main DBMS. I'm not certain that it does not work in PostgreSQL, but the descriptions I've seen elsewhere, as well as in this question, suggest that when the attempted INSERT means that any further activity in the transaction is doomed to fail too. Which is at best draconian and at worst 'unusable'.