Firebase child nodes are randomly deleted in random times
I have a firebase project with a lot of children per node (in the range of 100k+) and it uses real time database. The problem is, sometimes, completely randomly, few child items simply get removed.
For example, given the following database structure:
**Project** (root)
users
user1
user2
user3
.
.
.
user100000
one of the nodes among user1, user2... user100000 would just randomly get removed. This happens so randomly out of nowhere, I cannot reproduce it, and only affects less than 1% of the time, so out of more than 100k children, only 200ish would disappear in the span of 3 month. I went through every line of the code, but there is no function that removes a node. I only use updateChildValues, observe(.childAdded..., observeSingleEvent, and queryOrdered. There is no single method that starts with remove. Any ideas on why this is happening?
My first intuition is I am calling some write API like setValue or updateValue with Nil, but I am on Swift and that is not possible. I also looked at Firebase Web update() deletes all other child nodes, but this actually removes every child other than the one being added, which is definitely not happening to me. Has anyone also experienced this issue? Any help would be really appreciated. Thank you.
-----EDIT-----
Here are some example APIs that I use with write function. item variable is a dictionary defined by let.
let databaseRef = Database.database().reference()
newReferece = databaseRef.child("users").child(UID)
newReferece.setValue(item)
Another one:
theAPI.REF?.updateChildValues(dict, withCompletionBlock: { (error, ref) in
if error != nil {
onError(error!.localizedDescription)
} else {
onSuccess()
}
})
Related
This seems like it ought to have a very simple answer.
I am building an app using SwiftUI and firebase realtime database.
The database will have a node called items with a large number of children - on the order of 1,000.
I'd like the app to load the contents of that node children when it launches, then listen to firebase for future additions of children. Imagine the following:
struct Item { … } // some struct
var items: [Item] = []
let itemsRef = Database.database().reference().child("items")
According to firebase, this call ought to load in all the children one at a time, and then add new ones as they are added to the items node in firebase:
itemsRef.observe(.childAdded) { snapshot in
// add a single child based on snapshot
items.append(Item(fromDict: snapshot.value as! [String:String])
}
That gets the job done, but seems hugely inefficient compared to using the getData() method they supply, which hands us a dictionary containing all the children:
itemsRef.getData() { error, snapshot in
// set all children at once base on snapshot
items = Item.itemArray(fromMetaDict: snapshot.value as! [String:[String:String]])
}
It would seem best to use getData() initially, then observe(.childAdded) to monitor additions. But then how do we prevent the observe completion block from running 1,000 times when it fires up? The firebase docs say that that's what will happen:
This event is triggered once for each existing child and then again every time a new child is added to the specified path.
Thanks in advance!
PS I didn't think it necessary to include definitions for the functions Item.init(fromDict:) or Item.itemArray(fromMetaDict:) — hopefully it's clear what they are meant to do.
There is (or: should be) no difference between listening for .childAdded on a path versus listening for .value or calling `getData() on that same path. The distinction is purely client-side and the wire traffic is (or: should be) the same.
It is in fact quite common to listen for .child* events to manipulate some data structure/UI, and then also listen for the .value event to commit those changes, as .value is guaranteed to fire after all corresponding .child*.
A common trick to do the first big batch of data in .value or getData(), and then use .child* for granular updates is to have a boolean flag to indicate whether you got the initial data already, and set if to false initially.
In the .child* handlers, only process the data if the flag is true.
itemsRef.observe(.childAdded) { snapshot in
if isInitialDataProcessed {
items.append(Item(fromDict: snapshot.value as! [String:String])
}
}
And then in your .value/getData handler, process the data and then set the flag to true.
Goal:
My goal here is to fetch initial data, update UI after I am sure all data is fetched, and continue observing changes immediately:
The Problem
I see two ways to do this:
To use getData method and to fetch all data at once (bulky). This is ok cause I know I have fetched all data at once, and I can accordingly update UI and continue listening for changes (CRUD).
The problem with this approach is that I can't just attach listener after, to listen to a for new additions (inserts), and wait for new items. It works differently (which makes sense cause I fetched data without being in sync with a database through listeners), and immediately after I attach the listener, I get its callback triggered as many times as how many items are currently in a root node. So I am getting most likely the same data.
So this seems like overkill.
Second way to do this, is just to attach the listener, and get all data. But the problem with this is that I don't know when all data is fetched, cause it comes sequentially, one item by another. Thus I can't update UI accordingly.
Here are some code examples:
I am currently fetching all previous data with a getData method, like this:
func getInitialData(completion: #escaping DataReadCompletionHandler){
rootNodeReference.getData { optionalError, snapshot in
if let error = optionalError {
completion([])
return
}
if let value = snapshot.value,
let models = self.parseData(type: [MyModel].self, data: value) as? [MyModel]{
completion([MyModel](models.values))
}
}
}
As I said, with this, I am sure I have all previous data and I can set up my UI accordingly.
After this, I am interested in only new updates (updates, deletions, inserts).
And later I connect through listeners. Here is an example for a listener that listens when something new is added to a root node:
rootNodeReference.observe(DataEventType.childAdded, with: {[weak self] snapshot in
guard let `self` = self else {return}
if let value = snapshot.value,
let model = self.parseData(type: MyModel.self, data: value) as? MyModel{
self.firebaseReadDelegate?.event(type: .childAdded, model: model)
}
})
This would be great if with this listener I would somehow be able to continue only updates when something new is added.
Though, I guess option 2. would be a better way to go, but how to know when I have got all data through listeners?
There are two ways to do this, but they both depend on the same guarantee that Firebase makes about the order in which events are fired.
When you observe both child events and value events on the same path/query, the value event fires after all corresponding child events.
Because if this guarantee, you can add an additional listener to .value
rootNodeReference.observeSingleEvent(of: DataEventType.value, with: { snapshot in
... the initial data is all loaded
})
Adding the second listener doesn't increase the amount of data that is read from the database, because Firebase deduplicates them behind the scenese.
You can also forego the childAdded listener and just use a single observe(.value as shown in the documentation on reading a list by observing value events:
rootNodeReference.observe(.value) { snapshot in
for child in snapshot.children {
...
}
}
What I'm wanting to accomplish is save a score to firebase that has two values attached to it. Here's the code that writes the score to firebase.
func writeToFirebase() {
DispatchQueue.global(qos: .userInteractive).async {
self.ref = Database.database().reference()
if GameManager.instance.getTopScores().contains(GameManager.instance.getGameScore()) {
self.ref?.child("user")
.child(GameManager.instance.getUsername())
.child(String(GameManager.instance.getGameScore()))
.updateChildValues( [
"badge":GameManager.instance.getBadgeLevel(),
"vehicle": GameManager.instance.getVehicleSelected()
]
)
}
}
}
The issue I'm having is when a new score is saved with its values it sometimes overwrites the other scores. This seems to be random and its not when they're the same score or anything like that. Sometimes it will only overwrite one score and sometimes multiple. I'm watching firebase and I can see it being overwritten, it turns red and then is deleted. Sometimes the new score being added will be red and get deleted. The score doesn't need to be a child, but I don't know how to attach values to it if it's not. Any help is appreciated
This issue seems to happen occasionally so I am going to post my comment as an answer.
There are situations where an observer may be added to a node and when data changes in that node, like a write or update, it will fire that observer which may then overwrite the existing data with nil.
You can see this visually in the console as when the write occurs, you can see the data change/update, then it turns red and then mysteriously vanishes.
As suggested in my comment, add a breakpoint to the function that performs the write and run the code. See if that function is called twice (or more). If that's the case, the first write is storing the data properly but upon calling it a second time, the values being written are probably nil, which then makes the node 'go away' as Firebase nodes cannot exist without a value.
Generally speaking if you see your data turn red and vanish, it's likely caused by nil values being written to the node.
I have such a method in my Meteor app:
addLocation: (location, tid) => {
location = LocationSchema.clean(location);
LocationSchema.validate(location);
var lid = Locations.insert(location);
Thoughts.update({_id: tid}, {$push: {locations: lid}});
}
I subscribe both Locations and Thoughts collections. But unfortunately after calling my method only changes in Locations collection are visible - my modified thought is still the same and I must reload page to see its changes. Is it bug in Meteor or my mistake? Do you know some ways to solve or bypass this problem?
What's more, when I push some value to locations array in some thought with Robomongo, changes are visible though. It looks like the problem is that two changes try to be seen at the same time.
This bug has been busting me for the past 4 hours.
Also when I swap it round and get the User data first then Message data... the User.name will show, but the Message.message will not. So the data is definitely going in but the relationship between them seems to be broken.
Firstly, +1 for the effort with the image you created to illustrate your problem.
The cause of your issue is that you never assigned the user to the message (or vice-versa).
Try
message.fetchUser = user;
or
user.fetchMessage = message;
Then save your context and perform the fetch request.
As Rog said, I needed to assign the user to the message (or vice vera). Basically I placed this bit of code after where its entering the data.
messageDetails.fetchUser = userDetails