How can I filter the results on the Firestore Console by a timestamp field?
On the collection users, we have a field createdOn of type timestamp. If I want to filter the collection by field, I get the following dialog
I have tried entering the date as string
2019-09-15
2019-09-15T00:00:00Z
I have also tried using a timestamp as number in millis and seconds
1568505600000
1568505600
When looking at the requests sent to Firestore, the structured query uses a field filter with a value of corresponding to either stringValue or integerValue, but I think the timestampValue would be the right thing.
I do not want to adapt the query in the console and build my own requests. I know that there is always the option to sort the documents in the collection and then scroll to where it's interesting, but this will load all documents that are not relevant.
Is there a simple way to do what I want?
There is a new query builder tab in the console (I do not know when this was introduced, but I assume during the Firebase Summit 2022). For the query above, this would look like this
It even has a timestamp type in the select list.
I want to create some APIs, which let user pass in just strings for types like number, boolean. And automatically convert them before querying mongodb. Is it possible?
Yes, It is possible for MongoDB. You could write your own utility to convert string in mongo specific query or you could use some open source utilities like enter link description here.
Eventually, MongoDB accepts JSON string to execute the same client does also convert each query in the same JSON format. MongoDB clients or MongoDB doesn't need any predefined mapping or POJO.
This utility will convert string as shown below -
User string -
"select * from users where firstName='Vijay' AND lastName='Rajput'"
Then this utility will convert it into -
db.users.find({$and: [{firstName: 'Vijay'}, {lastName: 'Rajput'}]})
We were trying to index data from mongodb using logstash but we were unable to index array data type fields alone, also there were no errors in the log file.
It was an issue with MongoDB plugin which we used in logstash.
we added ruby code to the logstash-conf file to get the arrays from log_entry field.
Note: log_entry will have the complete field list and data.
when I try to write some values into my mongodb output in Pentaho, I would like null values of string fields to be translated to empty strings. Instead the key itself is not appearing in the mongo database. For example, if my field 'name' has a null value or a missing value, then I would like 'name':'' to appear in my mongo collection. Can anyone help me with this issue ?
Use the If field value is null step to convert null values to empty strings. To actually store the empty string in mongo set KETTLE_EMPTY_STRING_DIFFERS_FROM_NULL in kettle.properties
KETTLE_EMPTY_STRING_DIFFERS_FROM_NULL=Y
Another solution is to create a schema in mongo, to have default values, like #zydcom says in the comments.
My application generates logs in JSON format. The logs looks something like this :
{"LogLevel":"error","Datetime":"2013-06-21T11:20:17Z","Module":"DB","Method":"ExecuteSelect","Request":"WS_VALIDATE","Error":"Procedure or function 'WS_VALIDATE' expects parameter '#LOGIN_ID', which was not supplied."}
Currently, I'm pushing in the aforementioned log line as it is into mongoDB. But mongoDB stores the Datetime as a string (which is expected). Now that I want to run some data crunching job on these logs, I'd prefer to store the Datetime as mongoDB's native ISODate data type.
There are 3 ways I can think of for doing this :
i) parse every JSON log line and convert the string to ISODate type in the application code and then insert it. Cons : I'll have to parse each and every line before pushing it to mongoDB, which is going to be a little expensive
ii) After every insert run a query to convert the last inserted document's string date time to ISODate using
element.Datetime = ISODate(element.Datetime);
Cons : Again expensive, as I'm gonna be running one extra query per insert
iii) Modify my logs at generation point so that I don't have to do any parsing at application code level, or run an update query after every insert
Also, just curious, is there a way I can configure mongoDB to auto convert datetime strings to its native isodate format ?
TIA
EDIT:
I'm using pymongo for inserting the json logs
My file looks something like this :
{"LogLevel":"error","Datetime":"2013-06-21T11:20:17Z","Module":"DB","Method":"ExecuteSelect","Request":"WS_VALIDATE","Error":"Procedure or function 'WS_VALIDATE' expects parameter '#LOGIN_ID', which was not supplied."}
There are hundreds of lines like the one mentioned above.
And this is how I'm inserting them into mongodb:
for line in logfile:
collection.insert(json.loads(line))
The following will fix my problem:
for line in logfile:
data = json.loads(line)
data["Datetime"] = datetime.strptime(data["Datetime"], "%Y-%M-%DTHH:mmZ")
collection.insert(data)
What I want to do is get rid of the extra manipulation of datetime I'm having to do above. Hope this clarifies the problem.
Looks like you already have the answer... I would stick with:
for line in logfile:
data = json.loads(line)
data["Datetime"] = datetime.strptime(data["Datetime"], "%Y-%M-%DTHH:mmZ")
collection.insert(data)
I had a similar problem, but I didn't known beforehand where I should replace it by a datetime object. So I changed my json information to something like:
{"LogLevel":"error","Datetime":{"__timestamp__": "2013-06-21T11:20:17Z"},"Module":"DB","Method":"ExecuteSelect","Request":"WS_VALIDATE","Error":"Procedure or function 'WS_VALIDATE' expects parameter '#LOGIN_ID', which was not supplied."}
and parsed json with:
json.loads(data, object_hook=logHook)
with 'logHook' defined as:
def logHook(d):
if '__timestamp__' in d:
return datetime.strptime(d['__timestamp__'], "%Y-%M-%DTHH:mmZ")
return d
This logHook function could also be extended to replace many other 'variables' with elif, elif, ...
Hope this helps!
Also, just curious, is there a way I can configure mongoDB to auto convert datetime strings to its native isodate format ?
You probably want to create a Python datetime object for the timestamp, and insert that using PyMongo. This is stored under the hood as the native date object in MongoDB.
So, for example in Python:
from datetime import datetime
object_with_timestamp = { "timestamp": datetime.now() }
your_collection.insert(object_with_timestamp)
When this object gets queried from the Mongo shell, an ISODate object is present:
"timestamp" : ISODate("2013-06-24T09:29:58.615Z")
It depends on with what language/driver/utility you're pushing the log. I am assuming you're using mongoimport.
mongoimport doesn't support ISODate(). Refer to this issue https://jira.mongodb.org/browse/SERVER-5543 ISODate() is not a JSON format, hence not supported in mongoimport.
i) approach seems more efficient. ii) does two actions on mongo: insert & update. I had same issue while importing some log data into mongo. I ended up converting ISO 8601 format date to epoch format.
{"LogLevel":"error","Datetime":{"$date" : 1371813617000},"Module":"DB","Method":"ExecuteSelect","Request":"WS_VALIDATE","Error":"Procedure or function 'WS_VALIDATE' expects parameter '#LOGIN_ID', which was not supplied."}
Above JSON should work. Note that it is 64-bit not 32-bit epoch.