i have been working on a monitoring platform using logstash elasticsearch and kibana, the idea is to stream tweets and have a real time dashboard, everything here works just fine, now i want to store the tweets on mongodb and index them with elasticsearch, i've already done the first step of storing tweets, configuring mongodb as the output of logstash, but i can't seem to find a way to index this data with elasticsearch, i've looked for so many tutorials but i could't find a solution, if someone can help i would be really gratefull
Basically you need to:
Create an elastic search index
Read your tweets from mongodb
Index the tweets in elastic search
I'm guessing you already know how to do no. 2. For no. 1 you can either use the REST API, or any number of APIs available, depending on what the rest of your app is written in or what you are comfortable with.
For no. 3 is the same thing.
I'm using the Javascript API with this package.
Related
I've been doing research for the last several hours finding out how to do full text search in iOS apps with Firestore. I eventually realized full-text search is not supported. I read up on Algolia, watched tutorials, read articles and still don't see a way to push data from Firestore to Algolia in Swift. Can somebody guide me to a post or maybe add like a code snippet or something to show me how I can use this software? Thanks.
Algolia offers SWIFT API client
Swift API client is an open source, production-ready API wrapper that abstracts from the complexity of directly interfacing with the Algolia Search API
Basically to work with algolia using any API client, you have to do few things,
Install the algolia client
https://www.algolia.com/doc/api-client/getting-started/install/swift/?client=swift
Initialize the client with your application ID and KEY
https://www.algolia.com/doc/api-client/getting-started/instantiate-client-index/
Indexing your data & Setup searchable attributes
https://www.algolia.com/doc/api-client/methods/indexing/
https://www.algolia.com/doc/guides/managing-results/must-do/searchable-attributes/
Perform search
https://www.algolia.com/doc/api-reference/api-methods/search/
Many sub-actions available within these to improve your experience with Algolia. I recommend you to go through the docs fully.
Yes, there are few other ways available to work with algolia, like using cloud functions for indexing data into algolia and doing only search from your client. But as you specifically mentioned SWIFT the above references would help you to get started!
I want to connect mongodb with elasticsearch. I have used logstash-input-mongodb plugin, but this doesn't take my elasticsearch data updates. So when i update a document at mongodb it doesn't change at elasticsearch. I have searched through web but couldn't find a way to fix it? Is it a bug?
Which other plugins or tools do you suggest me to use for making the connection?
You could try mongoosastic however I think its community is a little bit small and there have been practically no updates since a year ago.
So I know the JDBC Rivers plugin is deprecated so even though it is being used I'd ideally not want to look at using something that is no longer supported.
However I have a few tables in a Postgres database with values that I need to be able to search in a Kibana view. Im new to the ELK stack but i've been messing around with some of their samples to get familiar.
I've seen some mentions of using Stored Procedures/Triggers from Postgres to send to Logstash. Although im not sure if this is the best way. Im not a developer but a QA so my coding skills are "ok" as im used to writing automation tests/etc...
What would be the best way to do this? I would want to probably capture updates to these tables (probably new inserts or updates) OR be able to poll the data every X period of time (30s or something). Lets pretend it's for a weather station and the tables contain humidity data from different weather sensors.
I'd want to be able to search in a Kibana view the Values/Station ID/etc...
Is this doable? Is there maybe a better way than using Triggers/Stored procedures?
I ended up using the JDBC driver and following https://www.elastic.co/blog/logstash-jdbc-input-plugin to get it moving and working (Which it does move). But it was a lot of setup for anyone that may see this answer.
I have been investigating a graph database and I have found neo4j and although this seems ideal I have also come across Mongodb.
Mongodb is not an official graph database but I wondered if it could be used for my scenario.
I am writing an application where users can have friends and those friends can have friends etc, the typical social part of a social network.
I was wondering in my situation whether Mongodb might suffice. How easy would it be to implement or do I really need to focus on REAL graph databases?
I do notice foursquare are using Mongodb so I presume it supports their infrastructure.
But how easy would it be to find all friends of my friends that also have friends in common, for example?
Although it wouldn't be impossible, MongoDB would not be a good fit for this scenario.
The reason is that MongoDB does not do JOINs. When you need a query which spans multiple documents, you need a separate query for each document.
In your example, each user document would have an array with the _id's of their friends. To find "all friends of the friends of UserA who are also friends of UserB" would mean that you would:
find userA and get his friends-array
find all users in that array and get their friend-arrays
find all users in these arrays who have UserB in their friends-array
These are three queries you have to perform. Between each of these queries, the result set has to be sent to the application, the application has to formulate a new query and send it back to the database. The result-set returned from the 2nd query can be quite large, which means that the 3rd query could take a while.
tl;dr: Use the right tool for the job. When your data is graph-based and you want to do graph-based queries on it, use a graph database.
You likely want an actual graph database as opposed to MongoDB. Try using the TinkerPop graph technology stack to get started. Using Blueprints (which is like JDBC for graphs) you can see the performance of MongoDB as a graph (using the Blueprints MongoDB implementation) versus Neo4j, Titan, or any number of other graph implementations.
I have a lots of documents stored on Amazon S3. My questions are:
Does Amazon provide any services/APIs using which I can index the contents of the document and search them (full text indexing and searching)?
If it does could someone please point me to any link in the documentation.
If it does not then could this be achieved with Lucene and Zend Framework? Have any one of you implemented this? Can I get some pointers?
UPDATE: I do not intend to save my index on Amazon S3 rather I am looking forward to indexing the contents of the documents on S3 and serving them based on a search.
You can see this question, or this blog post if you want to do pure lucene, or you can use Solr, which is probably easier. See also this post.
Zend has a PHP port of Lucene, which ties in very well. You can look at the Zend documentation for how to use it.