I've been using JetBrains Datagrip recently since I was able to get the whole suite for free. It's pretty nice, but I didn't notice any way to read GridFS using it. It seems like it should be common enough to have some sort of support, but I couldn't find any information online and its not immediately obvious from inside DataGrip.
We have created a feature request to implement GridFS support:
https://youtrack.jetbrains.com/issue/DBE-17458
Related
I've done quite a bit of searching, but haven't been able to find anything within this community that fits my problem.
I have a MongoDB collection that I would like to normalize and upload to Google Big Query. Unfortunately, I don't even know where to start with this project.
What would be the best approach to normalize the data? From there, what is recommended when it comes to loading that data to BQ?
I realize I'm not giving much detail here... but any help would be appreciated. Please let me know if I can provide any additional information.
If you're using python, easy way is to read collection chunky and use pandas' to_gbq method. Easy and quite fast to implement. But better to get more details.
Additionally to the answer provided by SirJ, you have multiple options to load data to BigQuery, including loading the data to Cloud Storage, local machine, Dataflow any more as mentioned here. Cloud Storage supports data in multiple formats such as CSV, JSON, Avro, Parquet and more. You also have various options to load data using Web UI, Command Line, API or using the Client Libraries which support C#, GO, Java, Node.JS, PHP, Python and Ruby.
I need to some visualize data from a Postgresql in Kibana. I have also ElasticSearch installed just in case. So how visualize data from a Postgresql in Kibana? Of course, I don't need the whole database, but only data returned by a custom sql query.
Also, I want it to be as simple as possible, I wouldn't like to use libraries I really don't need to use.
Kibana was built with Elastisearch in mind.
Having used it quite a lot in a startup I worked for, I can tell you that even the front-end query DSL (built on Lucene) will only work with Elasticsearch (or might need some serious tweaks).
I would advise you to push your data into Elasticsearch, and just work with Kibana the way it was made for :)
I started working on a small POC and decided to give Sails.js a try :)
Part of the POC we wanted to use DynamoDB since the project will eventually involve high scalability and we're not looking to hire full-time MongoDB expert at this point.
We used the module: https://github.com/gadelkareem/sails-dynamodb
Problem is there is no documentation and the module does not even work...
It seems the sails ORM is not ideal for DynamoDB and requires writing custom DB services. Does anyone have experience with this?
I was very excited to come across Sails but if it won't let us play nice with DynamoDB then it might very well be out as an option to us....
Anyone have experience with this or maybe something I'm missing?
One of the important plus of vogels is excellent documentation.
Sails-dynamodb adapter based on the vogels, but not all features are implemented in sails-dynamodb adapter. For example, vogels has Expression Filters.
Vogels able to create tables. Adapter can't. An adapter needs duplication table schema in sails files and dynamodb shell.
Vogels has some own types, such as uuid type, StringSet, NumberSet, TimeUUID. (Adapter can use it too, if includes Vogels and Joi lib)
Vogels and adapter have the same query (create, update, delete, find) capabilities.
Adapter allows without changing the code switch to another data base. Adapter encapsulates establishment of connection to database.
Conclusion - for most purposes this adapter is suitable for the work and do not need to work directly with the Vogels
Sails comes loaded with an ORM called "Waterline". There are some official waterline plugins such as mongodb, postgresql, mysql and then there are some unofficial ones created by the community. I'd assume right now that Dynamo is in the latter category since I have not come across it before. However, with that being said I would not take this experience as a reason to ditch Sails.js.
Sails.js is built with the intention that all of its components can be swapped out, this means you are not tied to a specific template engine, authentication libraries etc. and including your ORM choice.
Waterline is still being actively developed but it is sat at v0.12.1 as of writing this response. It isn't fully there yet so there will be the odd issues still around!
My recommendation? Take a look at swapping out waterline for a different ORM. Keep the flexibility Sails gives you and change out the component that doesn't meet your criteria. There are still many benefits to Sails you can utilise.
Vogels might be worth checking out: https://github.com/ryanfitz/vogels
Turning off waterline: Is there a way to disable waterline and use a different ORM in sails.js?
I've recently came across a need to store a higher amount of files in my application and because PaaS platform used to host the application provides mongo, I've would like to use it.
However because I'm quite inexperienced with mongo I have almost no idea what is the current state of mongo related plugins and tools for grails. What should I use? As I want to keep domain classes in SQL database and use mongo only to store related files (in this case it will be mostly a bunch of PDFs and text documents related to domain instance) the mongoDB ORM [1] plugin seems too "heavy". Unfortunately mongoDB ORM is probably the only mongo plugin for grails in active development at the moment.
In short, what would be the best plugin / library tool-set for this purpose? The closest thing that matches my need I've found is grails-mongo-files plugin [2], which is probably a little bit outdated with no further development.So far it seems that I will have to use mongo's java driver (or the gmongo wrapper) and write some storage service and taglib by myself (what is not necessary a bad thing).
[1] http://grails.org/plugin/mongodb
[2] https://github.com/quirklabs/grails-mongo-file
There is also the mongodb gridfs plugin. http://grails.org/plugin/mongodb-gridfs
One thing to consider is that gridfs effectively does two calls to mongo, one to retrieve file information and one to retrieve the file. So it might not be a good fit if your files are under 16 megabytes.
Here is a post on how to do this manually if you want to bypass plugins - http://jameswilliams.be/blog/entry/171
I am using Spring Data in my Java application to connect to MongoDb and have a requirement around versioning the documents (basically storing the history).
It seems that its pretty straightforward in Ruby, if one uses Mongoid
I was wondering if spring data has something similar for Java. Or are you better of trying to implement your own.
Yes there is a very good feature in Spring data which is auditing you can refer to the following link
http://www.javacodegeeks.com/2013/11/auditing-entities-in-spring-data-mongodb-2.html
After lot of research I found that https://javers.org/documentation/spring-boot-integration/. This works like rock solid and very easy to implement.
This library helps to store all the history of the changed fields and easy to query over it and it has great support of it. The sample POC shared here: https://nullbeans.com/auditing-using-spring-boot-mongodb-and-javers/