We have some tables in mongodb which we are extracting data and loading into mysql tables through kettle. How do I get the create timestamp of a record from mongodb in kettle?
I read some articles where we can get date from objectid but I was not able to extract in kettle.
can anyone give me the syntax for extracting date from objectid or is it possible to get hidden create ts from mongodb meta layer in kettle?
Thanks,
Deepthi
Related
This question already has an answer here:
Why does mongoose use schema when mongodb's benefit is supposed to be that it's schema-less?
(1 answer)
Closed 5 years ago.
I am a beginner with MongoDB and trying to learn MEAN Stack. So I am using Mongoose as the ORM
I read that MongoDB is a NoSQL database, but while using Mongoose as ORM, I am asked to create a schema first. Why is it so? There shouldn't be a schema ideally as MongoDB is a NoSQL database.
Thanks in advance.
Mongoose is an orm on top of mongodb , if you are using core mongodb you need not create any schema , you can just dump any data you want , but in mongoose you have a schema so that you can i have some basic key value pair for advanced searching and filtering and you can anytime update the schema. Or If you want to go schemaless and dump whatever the response is you can use a schema type like this var someSchema = {data:Object} and drop all your data in this data key and then you can easily extract whatever JSON data is inside your id field.
var mongoose = require('mongoose');
module.exports = mongoose.model('twitter', {
created_at:{
type:Date
},
dump:{
type:Object
}
});
In the above example dump is used to save whatever JSON I get as a response from twitter api and created_at contains only the creating date of tweet , so I have the entire data , but if i want to search tweets of a particular date I can search it using a find query on created_at and this query will be lot faster and here I have a fixed structure and a knowledge about what to expect of a find query each time a run one, So this is one of the benefit of using the mongoose orm i.e I don't lose data but I can maximise my searching ability by creating appropriate keys.
So basically mongoose is an ORM db , it offers you relational db features like creating foreign keys , not strictly foreign keys but you can create something like an id reference to another schema and later populate the field by the id associated parameters when you fetch data using your find query , also a relational schema is easy to manage , what mongoose does is it gives a JSON/BSON based db the power of relational db and you get best of both the world i.e you can easily maintain new keys or you don't need to worry about extracting each and every data from your operation and placing it properly/inserting it , you just need to see that your keys and values match , as well as you have flexibility in update operations while having a schema or table structure.
I am trying to query dates in mongodb.
The dates are stored as ISODate("2015-10-08T05:48:55.778+0000").
Now how should i do query like $gte or $lte.
I have been using Play plugin for reactive mongo
To query from the mongo shell, I would need to query with=>
{"endDateTime":{"$eq": new Date("2017-10-08T05:48:55.778+0000")}
OR,
{"endDateTime":{"$eq": ISODate("2017-10-08T05:48:55.778+0000")}
So, what should I do to query it using play reactive mongo. I have been using JodaTime. I am generating the Json Object of the query, and feeding to the find() api straightaway.
*Yes there a lot suggestion in SO, about the topic, but none of them seem to help me in this case. I could give more info if needed.
Update Answer:
Seems like I had some confusion, when converting the dates.
When I tried converting the String Date to Joda DateTime , the result when I print it in console, it would be shown as timestamp,but when I sent it to reactive mongo find it would convert to some form of string date "2015-10-08T05:48:55.778+0000".
So, I had to retrieve the millisecond conversion and send it to the respective api, and mongo would process without any issues.
I'm aware that the Mongo "ObjectId" has the method "getTimestamp()" , which works like
ObjectId("507f191e810c19729de860ea").getTimestamp()
And also I'm aware that it can be sorted based on built-in 'timestamp'
db.collection.find().sort({'timestamp': -1})
I know I can create a new field "created_time" in each document by converting ObjectId to created_time, then query based on this new field.
I've also read this post which converts the date range to ObjectId and then directly compare the ObjectId, but this method I'm worried about the other bytes which is not for time but for machine and process.
My question is, is there a way to directly query documents in a date range using Mongo built-in 'timestamp'? without extra field or extra effort.
something like below (but I tried below command and not working), which can directly query Mongo using its built-in timestamp.
db.collection.find({'timestamp':{$gt: new Date(ISODate("2015-08-14T14:00:00Z"))}})
I am trying to query out the data from my MongoDB database but there are some fields which I would like to omit as MongoDB will query the whole collections with id, n out.
I did this to limit the query but unfortunately only one field could be omitted but not the other which is the 'n' field. How can I omit two fields?
data = collection.find_one({"files_id": file_id},{"_id":0,"data":1})
And I also realized that my query for data has the field name (u'data') too, how can I query it so that it only returns the data? for this case it's a binary data
Example:
{u'data': Binary('\x00\x00\xed\x00\n\x00\x00\xd5\xa9\x00\x000\x00\x00\x00#\x00\x00\x0f\xff\xf0\x00\x0b\x80\x00\x00\x00
Kindly assist thanks!
I am migrating from MySQL to NoSql.Using mongo java driver I am able to insert the date fields which are currently being stored as NumberLong("1212121").However when I migrated the data from MySQL to NoSQL using Mongify the date field are getting saved as e.g ISODate("2015-04-19T18:48:38.121Z").To support the later I am trying through several options through SimpleDateFormat but could not get success to save the data in that ISODate format.
Please suggest.