I've been looking into NoSQL integration for Splunk using Hunk but there seems to be virtually no documentation on this process?
Is it possible? Are there docs?
I'm using Hunk 6.1
Specifically I am trying to use MongoDB as a backend datastore to splunk.
The short answer, is likely that there is no other NoSQL integrations out of the box and you'll have to roll your own for now.
Hunk has been marketed widely as Splunk for Hadoop and that is their first use case. (This can be seen in the Developer PDF still. General user documentation can be found http://docs.splunk.com/Documentation/Hunk/latest.
But the design of Hunk is that you define a "Virtual Index" that instead of connecting to and querying data from Splunk's built in indexes, you connect externally to another NoSQL data store. This requires a library to make requests to the NoSQL store and stream results back to Splunk (as can be seen in the diagram on this page)
I would suspect that they may expand the virtual index concept and ship libraries for other NoSQL datastores in the future as they have demand, but for now they have provided an example app that reportedly has documentation inside the app itself guide you on how to roll your own connecting library: http://apps.splunk.com/app/1802/
This functionality now exists with Hunk and the Hunk App for MongoDB. Here is the announcement : http://www.mongodb.com/presentations/splunks-hunk-powerful-way-visualize-your-data-stored-mongodb
The app is available here: https://apps.splunk.com/app/1810/
I hope that helps!
Related
I have a string for eg: "My name is John" stored in Firebase.
How would I query firebase so I can find all the posts in Firebase that have "John" ?
I can search for the first term in a string now using:
DataService.dataService.BASE_REF.child("Posts").child(selectedComment.commentKey).queryOrderedByChild("userComment").queryStartingAtValue(comment).queryEndingAtValue(comment+"\u{F8FF}").observeSingleEventOfType(.Value, withBlock: { (snapshot) in
where comment = "My"
I read about using Elastic search with Firebase but wanted to check if there was an easier way in Firebase before I looked at ElasticSearch/Flashlight for Firebase,
Unfortunately, Firebase doesn't support searching thru content like that (in any language SDK). From a Google Groups Post in July '16:
As a company that understands search, we're also a company that
understands using the best tool for the job. For fuzzy matching and
contains, a NoSQL, realtime data store isn't the correct tool--these
queries would be slow and scale poorly. BigQuery or ElasticSearch are
the right tool for providing useful results in a scalable and robust
manner.
Right now, this involves deploying a small node script to sync your
search results with the realtime data, as explained in the article
with the sample Flashlight lib. In the future, it will become more
"effortless" as we add integrations between Firebase and Cloud
products, particularly Cloud Functions and BigQuery interoperability.
BigQuery is, as I understand it, not specifically designed for user-facing search.
Elasticsearch (specifically, the Firebase plugin Flashlight) is a potential solution, but as you alluded to, it's an incredible amount of overhead (deploying/managing or renting an ES cluster, configuring the plugin, etc.). If content search is an important enough part of your app to justify that time/$, you may want to consider solutions beyond Firebase for your database needs, as it's by far one of the service's weakest areas.
In my opinion, you have a few options beyond Flashlight:
Algolia, a Search-as-a-service provider, does offer integration with Firebase, but I've never used it & so can't offer much more than to say that it exists.
Another alternative might be maintaining a collection of documents you want to search on another service, like AWS Cloud Search
Depending on the stage of your project & your needs, consider other Backends-as-a-Service that support more in terms of querying. E.g., GraphQL-as-a-service backends, like Scaphold.io, Graph.cool, and Reindex are all built on SQL databases, and (I believe) all support multiple types of querying.
I am trying to combined data from multiple sources like RDBMS, xml files, web services using Marklogic. For this as I see from MarkLogic documentation on Metadata Catalog (https://www.marklogic.com/solutions/metadata-catalog/), Data Virtualization (https://www.marklogic.com/solutions/data-virtualization/) and Data Unification it is very well possible. But I am not able to get hold of any documentation describing how exactly to go about it or which tools to use to achieve this.
Looking for some pointers.
As the second image in the data-virtualization link shows, you need to ingest all data into MarkLogic databases. MarkLogic can then be put in between to become the single entry point for end user applications that need access to that data.
The first link describes the capabilities of MarkLogic to hold all kinds of data. It partly does so by storing them as-is, partly by extracting text and metadata for searching, partly by conversion (if you needs go beyond what the original format allows).
MarkLogic provides the general purpose MarkLogic Content Pump (MLCP) tool for this purpose. It allows ingesting zipped or unzipped files, and applying transformations if necessary. If you need to retrieve your data from a different database, you might need a bit more work to get that out. http://developer.marklogic.com holds tutorials, blogs, and tools that should help you get going. Searching the MarkLogic Mailing List through http://marklogic.markmail.org/ can provide answers as well.
HTH!
Combining a lot of data is a very broad topic. Can you describe a couple types of data you'd like to integrate, and what services or queries you would like to build on that data?
We have an EZ Publish 5 already operational with an Oracle 11g database as a persistence backend.
We planned to get rid of Oracle and we know that we could use Postgresql for sure as a new RDBMS, but we also planned to use MongoDB whenever it's possible.
So the question is: Can we pair EZ Publish 5 to a MongoDB NoSQL database ?
Quick answer: You cannot now as there is no MongoDB driver implemented.
There was a NoSQL Driver planned in the 5.x series, but it is not yet implemented in the latest eZ Publish 2013.06.
With the official roadmap not being updated since the 4.6 release, it hard to guess when the NoSQL driver will be implemented, and if MongoDB will be supported.
Theorically, as the new persistence API make it "easier" to add new storage drivers, you could write a custom MongoDB storage driver.
But in practice implementing a storage driver is quite complex and resource consuming.
Contacting the eZ Systems office in your region and directly ask them when a MongoDB driver will be available might be fastest way to get a clear answer.
I hope it helps.
It is not currently possible as there is no storage engine implemented for MongoDB.
It is indeed planned to implement a NoSQL engine, but there is no version tag on this for now.
However, the new persistence API makes it possible to "easily" implement such a storage engine, but you will be limited by the fact that eZ Publish 5 still uses the legacy back office which runs in the legacy stack (so with 4.x infrastructure which is not compatible with NoSQL).
In short: Implementing a new storage engine will make it only accessible from Public API and REST API.
Since ez5 is a symfony application you can achieve this the symfony way
http://symfony.com/doc/current/bundles/DoctrineMongoDBBundle/index.html
just remember they have renamed the app folder and the appkernel.php.
Haven't tried it yet though
now I have an iPhone App and basically I want to exchange data from my database (MongoDB) on a server.
Could you please tell me exactly what I should do?
Forgive my innocence, I am a beginner in this area...Thank you very much ahead of time!!
I think you have two options to talk to mongo :
1) Use the rest interface http://www.mongodb.org/display/DOCS/Http+Interface
2) Use Objective-C driver: https://github.com/timburks/NuMongoDB
If you're not completely tied to MongoDB, have a look at CouchDB. It's essentially the same thing as MongoDB (JSON document store) but for the web. They have a nice built-in REST interface which makes database interaction in mobile/server environments very nice.
http://couchdb.apache.org/
In addition to Sid's options, you can also build your own backend that talks to mongodb, that communicates via REST (in your language of choice). This way you can pool your connections on the backend and avoid connectivity issues from the devices.
I am looking for an API that performs functionality roughly analogous to Rackspace Cloud Files / OpenStack Swift, Microsoft Azure Blob Storage, or Amazon S3 that can be run on a Windows Server.
I am not speaking of all the add-ons including replication, etc, but an API that enables a similar RESTful API for the storage/serving (including Anonymous). Some examples of functionality I like, and would be missing if I rolled my own right now, are:
Rackspace's Large Files support.
Amazon S3's Root Document support.
Microsoft Azure BLOB storage Page Blobs and Authentication.
Options like MongoDB's GridFS are getting close, but wouldn't quite cut it. RavenDB's "Attachments" functionality is pretty close, I understand it only supports up to 2Gb via the ESENT storage engine
Just to clarify, I'm not exactly sure what form this would take. I'm not looking for a pre-built product (which I don't see exists), but perhaps a stub of a project, an open source project planning to provide this functionality, people who might have developed their own similar solution in C#, etc.
We have RavenFS that handles that scenario, I think.
It is a commercial offering, though.