yes I just started to understand MongoDB. Therefore I have installed MongoDB in the folder "Program files" yesterday and tryed to figure out, how to work with it.
Today I simply tryed to start it again. But something went wrong. To understand the beginning better I deinstalled MongoDB, deleted the MongoDB, data, log folders and have installed MongoDB again direct on c:/.
Started the server:
mongod --directoryperdb --dbpath c:\mongodb\data\db --logpath c:\mongodb\log\mongo.log --logappend --auth --rest --install
Started the service:
net start MongoDB
And the Mongo shell:
mongo
Now I try "show dbs" with the following result:
>show dbs
2017-09-14T10:41:49.541+0200 E QUERY [thread1] Error: listDatabases failed:{
"ok" : 0,
"errmsg" : "not authorized on admin to execute command { listDatabases: 1.0 }",
"code" : 13,
"codeName" : "Unauthorized"
} :
_getErrorWithCode#src/mongo/shell/utils.js:25:13
Mongo.prototype.getDBs#src/mongo/shell/mongo.js:62:1
shellHelper.show#src/mongo/shell/utils.js:769:19
shellHelper#src/mongo/shell/utils.js:659:15
#(shellhelp2):1:1
More I haven't done already. What can I do to show the dbs?
Of cource, I have tryed the learnings of the following post to, without success:
show dbs gives "Not Authorized to execute command" error
Looks like you are starting the db with --auth but not connecting with username/password. If you have not created any user then remove --auth and restart db. Then you will be able to connect.
I want to setup on my dev machine a mongodb replication, same machine running mongodb instances on docker.
so i have docker containers running mongodb instances on them, working, fine and setup.
// some images omitted
$ docker ps -a
0.0.0.0:49157->27017/tcp
0.0.0.0:49156->27017/tcp
0.0.0.0:49155->27017/tcp
then, on my PRIMARY, i tried adding something
PRIMARY> rs.add("172.17.0.141:27017")
{
"ok" : 0,
"errmsg" : "need most members up to reconfigure, not ok : 172.17.0.141:27017",
"code" : 13144
}
//===============
PRIMARY> rs.add("0.0.0.0:27017")
{
"ok" : 0,
"errmsg" : "need most members up to reconfigure, not ok : 0.0.0.0:27017",
"code" : 13144
}
how do i add mongodb instances inside containers to replication using docker?
nvm, i solved my question.
from this post, http://version2beta.com/articles/mongod-replica-set-slave/, i give credit to the author.
i have disabled my journal to my slaves
storage:
smallFiles: true
journal:
enabled: false
then get the IP from container
$ docker inspect e123
then in mongo-cli
PRIMARY> rs.add("172.17.0.150:27017")
I have a problem removing records from a capped collection. DB is complaining that command doesn't exists.
There is a collection that is capped. I have verified by issuing isCapped().
I switch to a db that contains the collection.
Then I run
db.runCommand({ emptycapped: 'events'})
as a result I get :
{
"ok" : 0,
"errmsg" : "no such cmd: emptycapped",
"bad cmd" : {
"emptycapped" : "events"
}
}
Environment details :
MongoDB shell version: 2.2.0
Mongod Server version: 2.4.1 on Ubuntu 12.04
Any ideas what could be the possible cause ?
As Enrique Fueyo's commented:
In the docs (http://docs.mongodb.org/manual/reference/command/emptycapped) you can read that "...is not enabled by default. emptycapped must be enabled by using --setParameter enableTestCommands=1 on the mongod command line."
You need to start mongo with a command such this:
$ mongod --config /usr/local/etc/mongod.conf --setParameter enableTestCommands=1
I have gone through many blogs and sites about configuring Elasticsearch for MongoDB to index Collections in MongoDB but none of them were straightforward.
Please explain to me a step by step process for installing elasticsearch, which should include:
configuration
run in the browser
I am using Node.js with express.js, so please help accordingly.
This answer should be enough to get you set up to follow this tutorial on Building a functional search component with MongoDB, Elasticsearch, and AngularJS.
If you're looking to use faceted search with data from an API then Matthiasn's BirdWatch Repo is something you might want to look at.
So here's how you can setup a single node Elasticsearch "cluster" to index MongoDB for use in a NodeJS, Express app on a fresh EC2 Ubuntu 14.04 instance.
Make sure everything is up to date.
sudo apt-get update
Install NodeJS.
sudo apt-get install nodejs
sudo apt-get install npm
Install MongoDB - These steps are straight from MongoDB docs.
Choose whatever version you're comfortable with. I'm sticking with v2.4.9 because it seems to be the most recent version MongoDB-River supports without issues.
Import the MongoDB public GPG Key.
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
Update your sources list.
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list
Get the 10gen package.
sudo apt-get install mongodb-10gen
Then pick your version if you don't want the most recent. If you are setting your environment up on a windows 7 or 8 machine stay away from v2.6 until they work some bugs out with running it as a service.
apt-get install mongodb-10gen=2.4.9
Prevent the version of your MongoDB installation being bumped up when you update.
echo "mongodb-10gen hold" | sudo dpkg --set-selections
Start the MongoDB service.
sudo service mongodb start
Your database files default to /var/lib/mongo and your log files to /var/log/mongo.
Create a database through the mongo shell and push some dummy data into it.
mongo YOUR_DATABASE_NAME
db.createCollection(YOUR_COLLECTION_NAME)
for (var i = 1; i <= 25; i++) db.YOUR_COLLECTION_NAME.insert( { x : i } )
Now to Convert the standalone MongoDB into a Replica Set.
First Shutdown the process.
mongo YOUR_DATABASE_NAME
use admin
db.shutdownServer()
Now we're running MongoDB as a service, so we don't pass in the "--replSet rs0" option in the command line argument when we restart the mongod process. Instead, we put it in the mongod.conf file.
vi /etc/mongod.conf
Add these lines, subbing for your db and log paths.
replSet=rs0
dbpath=YOUR_PATH_TO_DATA/DB
logpath=YOUR_PATH_TO_LOG/MONGO.LOG
Now open up the mongo shell again to initialize the replica set.
mongo DATABASE_NAME
config = { "_id" : "rs0", "members" : [ { "_id" : 0, "host" : "127.0.0.1:27017" } ] }
rs.initiate(config)
rs.slaveOk() // allows read operations to run on secondary members.
Now install Elasticsearch. I'm just following this helpful Gist.
Make sure Java is installed.
sudo apt-get install openjdk-7-jre-headless -y
Stick with v1.1.x for now until the Mongo-River plugin bug gets fixed in v1.2.1.
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb
sudo dpkg -i elasticsearch-1.1.1.deb
curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
sudo mv *servicewrapper*/service /usr/local/share/elasticsearch/bin/
sudo rm -Rf *servicewrapper*
sudo /usr/local/share/elasticsearch/bin/service/elasticsearch install
sudo ln -s `readlink -f /usr/local/share/elasticsearch/bin/service/elasticsearch` /usr/local/bin/rcelasticsearch
Make sure /etc/elasticsearch/elasticsearch.yml has the following config options enabled if you're only developing on a single node for now:
cluster.name: "MY_CLUSTER_NAME"
node.local: true
Start the Elasticsearch service.
sudo service elasticsearch start
Verify it's working.
curl http://localhost:9200
If you see something like this then you're good.
{
"status" : 200,
"name" : "Chi Demon",
"version" : {
"number" : "1.1.2",
"build_hash" : "e511f7b28b77c4d99175905fac65bffbf4c80cf7",
"build_timestamp" : "2014-05-22T12:27:39Z",
"build_snapshot" : false,
"lucene_version" : "4.7"
},
"tagline" : "You Know, for Search"
}
Now install the Elasticsearch plugins so it can play with MongoDB.
bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.6.0
bin/plugin --install elasticsearch/elasticsearch-mapper-attachments/1.6.0
These two plugins aren't necessary but they're good for testing queries and visualizing changes to your indexes.
bin/plugin --install mobz/elasticsearch-head
bin/plugin --install lukas-vlcek/bigdesk
Restart Elasticsearch.
sudo service elasticsearch restart
Finally index a collection from MongoDB.
curl -XPUT localhost:9200/_river/DATABASE_NAME/_meta -d '{
"type": "mongodb",
"mongodb": {
"servers": [
{ "host": "127.0.0.1", "port": 27017 }
],
"db": "DATABASE_NAME",
"collection": "ACTUAL_COLLECTION_NAME",
"options": { "secondary_read_preference": true },
"gridfs": false
},
"index": {
"name": "ARBITRARY INDEX NAME",
"type": "ARBITRARY TYPE NAME"
}
}'
Check that your index is in Elasticsearch
curl -XGET http://localhost:9200/_aliases
Check your cluster health.
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
It's probably yellow with some unassigned shards. We have to tell Elasticsearch what we want to work with.
curl -XPUT 'localhost:9200/_settings' -d '{ "index" : { "number_of_replicas" : 0 } }'
Check cluster health again. It should be green now.
curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
Go play.
Using river can present issues when your operation scales up. River will use a ton of memory when under heavy operation. I recommend implementing your own elasticsearch models, or if you're using mongoose you can build your elasticsearch models right into that or use mongoosastic which essentially does this for you.
Another disadvantage to Mongodb River is that you'll be stuck using mongodb 2.4.x branch, and ElasticSearch 0.90.x. You'll start to find that you're missing out on a lot of really nice features, and the mongodb river project just doesn't produce a usable product fast enough to keep stable. That said Mongodb River is definitely not something I'd go into production with. It's posed more problems than its worth. It will randomly drop write under heavy load, it will consume lots of memory, and there's no setting to cap that. Additionally, river doesn't update in realtime, it reads oplogs from mongodb, and this can delay updates for as long as 5 minutes in my experience.
We recently had to rewrite a large portion of our project, because its a weekly occurrence that something goes wrong with ElasticSearch. We had even gone as far as to hire a Dev Ops consultant, who also agrees that its best to move away from River.
UPDATE:
Elasticsearch-mongodb-river now supports ES v1.4.0 and mongodb v2.6.x. However, you'll still likely run into performance problems on heavy insert/update operations as this plugin will try to read mongodb's oplogs to sync. If there are a lot of operations since the lock(or latch rather) unlocks, you'll notice extremely high memory usage on your elasticsearch server. If you plan on having a large operation, river is not a good option. The developers of ElasticSearch still recommend you to manage your own indexes by communicating directly with their API using the client library for your language, rather than using river. This isn't really the purpose of river. Twitter-river is a great example of how river should be used. Its essentially a great way to source data from outside sources, but not very reliable for high traffic or internal use.
Also consider that mongodb-river falls behind in version, as its not maintained by ElasticSearch Organization, its maintained by a thirdparty. Development was stuck on v0.90 branch for a long time after the release of v1.0, and when a version for v1.0 was released it wasn't stable until elasticsearch released v1.3.0. Mongodb versions also fall behind. You may find yourself in a tight spot when you're looking to move to a later version of each, especially with ElasticSearch under such heavy development, with many very anticipated features on the way. Staying up on the latest ElasticSearch has been very important as we rely heavily on constantly improving our search functionality as its a core part of our product.
All in all you'll likely get a better product if you do it yourself. Its not that difficult. Its just another database to manage in your code, and it can easily be dropped in to your existing models without major refactoring.
River is a good solution once you want to have a almost real time synchronization and general solution.
If you have data in MongoDB already and want to ship it very easily to Elasticsearch like "one-shot" you can try my package in Node.js https://github.com/itemsapi/elasticbulk.
It's using Node.js streams so you can import data from everything what is supporting streams (i.e. MongoDB, PostgreSQL, MySQL, JSON files, etc)
Example for MongoDB to Elasticsearch:
Install packages:
npm install elasticbulk
npm install mongoose
npm install bluebird
Create script i.e. script.js:
const elasticbulk = require('elasticbulk');
const mongoose = require('mongoose');
const Promise = require('bluebird');
mongoose.connect('mongodb://localhost/your_database_name', {
useMongoClient: true
});
mongoose.Promise = Promise;
var Page = mongoose.model('Page', new mongoose.Schema({
title: String,
categories: Array
}), 'your_collection_name');
// stream query
var stream = Page.find({
}, {title: 1, _id: 0, categories: 1}).limit(1500000).skip(0).batchSize(500).stream();
elasticbulk.import(stream, {
index: 'my_index_name',
type: 'my_type_name',
host: 'localhost:9200',
})
.then(function(res) {
console.log('Importing finished');
})
Ship your data:
node script.js
It's not extremely fast but it's working for millions of records (thanks to streams).
Here I found another good option to migrate your MongoDB data to Elasticsearch.
A go daemon that syncs mongodb to elasticsearch in realtime.
Its the Monstache. Its available at : Monstache
Below the initial setp to configure and use it.
Step 1:
C:\Program Files\MongoDB\Server\4.0\bin>mongod --smallfiles --oplogSize 50 --replSet test
Step 2 :
C:\Program Files\MongoDB\Server\4.0\bin>mongo
C:\Program Files\MongoDB\Server\4.0\bin>mongo
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.0.2
Server has startup warnings:
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2019-01-18T16:56:44.931+0530 I CONTROL [initandlisten]
MongoDB Enterprise test:PRIMARY>
Step 3 : Verify the replication.
MongoDB Enterprise test:PRIMARY> rs.status();
{
"set" : "test",
"date" : ISODate("2019-01-18T11:39:00.380Z"),
"myState" : 1,
"term" : NumberLong(2),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"appliedOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"durableOpTime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
}
},
"lastStableCheckpointTimestamp" : Timestamp(1547811517, 1),
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 736,
"optime" : {
"ts" : Timestamp(1547811537, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2019-01-18T11:38:57Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1547810805, 1),
"electionDate" : ISODate("2019-01-18T11:26:45Z"),
"configVersion" : 1,
"self" : true,
"lastHeartbeatMessage" : ""
}
],
"ok" : 1,
"operationTime" : Timestamp(1547811537, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1547811537, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
MongoDB Enterprise test:PRIMARY>
Step 4.
Download the "https://github.com/rwynn/monstache/releases".
Unzip the download and adjust your PATH variable to include the path to the folder for your platform.
GO to cmd and type "monstache -v"
# 4.13.1
Monstache uses the TOML format for its configuration. Configure the file for migration named config.toml
Step 5.
My config.toml -->
mongo-url = "mongodb://127.0.0.1:27017/?replicaSet=test"
elasticsearch-urls = ["http://localhost:9200"]
direct-read-namespaces = [ "admin.users" ]
gzip = true
stats = true
index-stats = true
elasticsearch-max-conns = 4
elasticsearch-max-seconds = 5
elasticsearch-max-bytes = 8000000
dropped-collections = false
dropped-databases = false
resume = true
resume-write-unsafe = true
resume-name = "default"
index-files = false
file-highlighting = false
verbose = true
exit-after-direct-reads = false
index-as-update=true
index-oplog-time=true
Step 6.
D:\15-1-19>monstache -f config.toml
I found mongo-connector useful. It is form Mongo Labs (MongoDB Inc.) and can be used now with Elasticsearch 2.x
Elastic 2.x doc manager: https://github.com/mongodb-labs/elastic2-doc-manager
mongo-connector creates a pipeline from a MongoDB cluster to one or more target systems, such as Solr, Elasticsearch, or another MongoDB cluster. It synchronizes data in MongoDB to the target then tails the MongoDB oplog, keeping up with operations in MongoDB in real-time. It has been tested with Python 2.6, 2.7, and 3.3+. Detailed documentation is available on the wiki.
https://github.com/mongodb-labs/mongo-connector
https://github.com/mongodb-labs/mongo-connector/wiki/Usage%20with%20ElasticSearch
Here how to do this on mongodb 3.0. I used this nice blog
Install mongodb.
Create data directories:
$ mkdir RANDOM_PATH/node1
$ mkdir RANDOM_PATH/node2>
$ mkdir RANDOM_PATH/node3
Start Mongod instances
$ mongod --replSet test --port 27021 --dbpath node1
$ mongod --replSet test --port 27022 --dbpath node2
$ mongod --replSet test --port 27023 --dbpath node3
Configure the Replica Set:
$ mongo
config = {_id: 'test', members: [ {_id: 0, host: 'localhost:27021'}, {_id: 1, host: 'localhost:27022'}]};
rs.initiate(config);
Installing Elasticsearch:
a. Download and unzip the [latest Elasticsearch][2] distribution
b. Run bin/elasticsearch to start the es server.
c. Run curl -XGET http://localhost:9200/ to confirm it is working.
Installing and configuring the MongoDB River:
$ bin/plugin --install
com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb
$ bin/plugin --install elasticsearch/elasticsearch-mapper-attachments
Create the “River” and the Index:
curl -XPUT 'http://localhost:8080/_river/mongodb/_meta' -d '{
"type": "mongodb",
"mongodb": {
"db": "mydb",
"collection": "foo"
},
"index": {
"name": "name",
"type": "random"
}
}'
Test on browser:
http://localhost:9200/_search?q=home
Since mongo-connector now appears dead, my company decided to build a tool for using Mongo change streams to output to Elasticsearch.
Our initial results look promising. You can check it out at https://github.com/electionsexperts/mongo-stream. We're still early in development, and would welcome suggestions or contributions.
Consider, I have 4 replicate sets and the config is as follows:
{
"_id": "rs_0",
"version": 5,
"members" : [
{"_id": 1, "host": "127.0.0.1:27001"},
{"_id": 2, "host": "127.0.0.1:27002"},
{"_id": 3, "host": "127.0.0.1:27003"},
{"_id": 4, "host": "127.0.0.1:27004"}
]
}
I am able to connect to all sets using
mongo --port <port>
There are documents for getting information on Convert a Standalone to a Replica Set, but can anyone tell me how to convert back to standalone from replica set?
Remove all secondary hosts from replica set (rs.remove('host:port')), restart the mongo deamon without replSet parameter (editing /etc/mongo.conf) and the secondary hosts starts in standalone mode again.
The Primary host is tricky one, because you can't remove it from the replica set with rs.remove.
Once you have only the primary node in the replica set, you should exit mongo shell and stop mongo. Then you edit the /etc/mongo.conf and remove the replSet parameter and start mongo again.
Once you start mongo you are already in standalone mode, but the mongo shell will prompt a message like:
2015-07-31T12:02:51.112+0100 [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
to remove the warning you can do 2 procedures:
1) Droping the local db and restarting mongo:
use local
db.dropDatabase();
/etc/init.d/mongod restart
2)Or if you don't want to be so radical, you can do:
use local
db.system.replset.find()
and it will prompt a message like:
{ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] }
then you will erase it using:
db.system.replset.remove({ "_id" : "replicaSetName", "version" : 1, "members" : [ { "_id" : 0, "host" : "hostprimary:mongoport" } ] })
and it will probably prompt:
WriteResult({ "nRemoved" : 1 })
Now, you can restart the mongo and the warning should be gone, and you will have your mongo in standalone mode without warnings
Just remove a host from replica set (rs.remove('host:port')), relaunch it without replSet parameter and it's standalone again.
On an Ubuntu Machine
Stop your mongo server
open /etc/mongod.conf
Comment the replication and replSetName line
#replication:
#replSetName: rs0
Start your mongo server and go to mongo shell
drop local database
use local
db.dropDatabase()
Restart mongo
The MongoDB Documentation suggests the following to perform maintenance on a replica set member, which brings the the replica set member into standalone mode for further operations. With little modification it can be made standalone:
If node in concern is the only node in a shard, drain the chunks to other shards as per MongoDB documentation here, or else the sharded database will break, i.e.
Make sure balancer is enabled by connecting to mongos and run sh.startBalancer(timeout, interval)
For the shard in concern, go to admin database and db.adminCommand( { removeShard: "mongodb0" } )
Check draining status by repeating above removeShard command, wait for draining to complete
If node in concern is primary, do rs.stepDown(300)
Stop the node by running db.shutdownServer()
Change the yaml config by:
commenting out replication.replSetName (--replSetName in command line)
commenting out sharding.clusterRole for shard or config server (--shardsvc and --configsvr in command line)
commenting out net.port, then change it to a different port (--port in command line)
Start the mongod instance
If change is permanent, go to other mongod instance and run rs.remove("host:port")
After this, the node in concern should be up and running in standalone mode.
Follow below steps :
Go to mongo shell on Secondary servers
Stop the secondary servers by using below command :
use admin
db.shutdownServer()
Go to Linux shell- on secondary servers and type below command :
sudo service mongod stop
Starting the MongoDB replication -
Go to Linux shell - on secondary servers and type below command :
sudo service mongod start
Starting the MongoDB replication -
Go to primary and type below commands to start the replication :
a] rs.initiate()
b] rs.add("Secondar -1:port no")
c] rs.add("Secondary-2:port no")
d] rs.add({ "_id" : 3, "host" : "Hidden_member:port no", "priority" : 0,
"hidden" : true })
e] rs.status()