I have a large query running on a collection pulling back the date in NumberLong format, I then export to a .csv where I run a formula to convert the date to a format a human can use. Scrolling the web I am not able to find a clear way of pulling the date in 'YYY-MM-DD' format and not the NumberLong on the fly. Has anyone managed this?
If you are doing this as a once off action, one way of doing this is to use the forEach() method to convert the date and inject the new converted object into a temporary collection. Then dump that collection to CSV.
Something like this:
db.myCollection.find().forEach(function(element) {
element.dateField = new Date(element.dateField);
db.myTempCollection.insert(element);
});
Otherwise, I recommend using your programming language of choice and implementing a program or script to do the conversion.
Related
I would be grateful if someone could explain here step by step what the process of changing the date format and column type from string to date should look like in the table imported via Hive View to HDP 2.6.5.
The data source is the well-known MovieLens 100K Dataset set ('u.item' file) from:
https://grouplens.org/datasets/movielens/100k/
$ hive --version is: 1.2.1000.2.6.5.0-292
Date format for the column is: '01-Jan-1995'
Data type of column is: 'string'
ACID Transactions is 'On'
Ultimately, I would like to convert permanently the data in the entire column to the correct Hive format 'yyyy-MM-dd' and next column type to 'Date'.
I have looked at over a dozen threads regarding similar questions before. Of course, the problem is not to display the column like this, it can be easily done using just:
SELECT from_unixtime(unix_timestamp(prod_date,'dd-MMM-yyyy'),'yyyy-MM-dd') FROM moviesnames;
The problem is to finally write it down this way. Unfortunately, this cannot be done via UPDATE in the following way, despite the inclusion of atomic operations in Hive config.
UPDATE moviesnames SET prodate = (select to_date(from_unixtime(UNIX_TIMESTAMP(prod_date,'dd-MMM-yyyy'))) from moviesnames);
What's the easiest way to achieve the above using Hive-SQL? By copying and transforming a column or an entire table?
Try this:
UPDATE moviesnames SET prodate = to_date(from_unixtime(UNIX_TIMESTAMP(prod_date,'dd-MMM-yyyy')));
everyone. A little bit of help would be nice, and I thank you for it. I'm trying to save a document which contains a datetime field. Using mongodbspark connector through MongoSpark.save() method, it could be a challenge:
if a set the field as a string, it's quite obvious that what will be saved is a string, not an isodate (even if the string fulfilled the ISO 8601 format
if I build an expression like this: my:date:{$date:}, where xxxx is some epoch time in milliseconds, then I get this BulkWriteError which set that '$' sign is not valid for storage
I get documents to be update from a library which returns BsonDocument docs. Datetime fields are treated like BsonDateTime fields, so I need to make some conversions before saving/updating 'cause getting the corresponding json string fro the BsonDocument, generates the $date non-valid-for-storage stuff.
For obtaind the BsonDocument, I just called a method from a library built by another developer:
val bdoc = handlePermanentProduct(p_id, operationsByProduct)
Then I convert the org.bson.BsonDocument in a org.bson.Document using a method I wrote:
val doc: Document = convert(bdoc)
Then, the usual code for getting a dataframe & saving/updating my documents
val docs = sc.parallelize(Seq(doc.toJson))
val df = sparkSession.read.json(rdd)
MongoSpark.save(df.write.option("collection", "products").option("replaceDocument", "false").mode(SaveMode.Append))
Thanks again, and in advance
I'm using Scala 2.11.8, Spark 2.11, and Mongodb Spark connect v2.1
Definitively, the way I was trying to use for saving/updating is not the right way. I found out, reading the documentation, of course, that there is a type matching process when I want to save/update using MongoSpark.save(...) method: datetime fields can be creates as java.sql.Timestamp, so driver makes the proper conversions. It was really easy once I found that. So, it's solved.
How can I write the serverValue.timestamp() as a separate child in Firebase?
I want to be able to write something like this:
-Country
---Location
------serverValue.timestamp()
---------Name: Peter
---------Age: 30
---------Class: Physics
When I try this it fails:
self.ref?.child(country).child(location).child(serverValue.timestamp()).updateChildValues(["Name:" : name, "Age:" : age, "Class:" : class])
I can't seem to get it right...
According to the docs timeStamp is a placeholder for a value in which the database recognize and put in the current servertime. This does not apply for Strings in the database. Strings in the databases are names for folders, documents, collections, keys etc. So the method you are calling only works for a value in combination with a key, for storing data in JSON format.
If you REALLY want to use the current timestamp without users faking the current time, I think the only thing you can do is:
write the data to a temp folder
trigger a Cloud Function that listens to that folder/document
write the data back to your preferred folder/document, while the name of the folder/document is Date.now() (function in javascript)
Else you could always use Swift's current timestamp: How to get 18-digit current timestamp in Swift?
I have the oracle database field value "11-JUL-16 02.51.45.000000000 AM" for field date_updated.
When I retrieve records via orm lite query and iterate over the result set .. I get the data in this format "2016-7-11.2.51. 45. 0" where the java pojo object mapping field is of type String.
Aim is to update these timestamps after processing them. I am not able to covert the date to update(parse error) or retrieve the date as is.
Searched allover but couldn't find an answer. I tried changing the pojo field type to Date/Timestamp(sql) but couldn't get it to work. Any Help would really appreciate ..
I am now building an iPhone app and it involves core data. One of the entities has an attribute with Date type, which effectively generates a column with TIMESTAMP type in the corresponding sqlite DB. The value looks something like 320928592.400471
My question is... how can I convert ordinary datetime into the TIMESTAMP type? I would like to preload some static data to the DB. Therefore, I need to know how to store the data directly to the DB.
Chances are that number is the same number returned by NSDate's timeIntervalSinceReferenceDate, i.e. seconds since 1 January 2001.
It might be easier to either populate the database on the first run of your program, or to generate the prefilled database and export it from your phone to include in the bundle.