I'm looking for a little direction on how to set up services on AWS. I have an application that is build with Node.js and uses mongodb (and mongoose as the ODM). I'm porting everything over to AWS and would like to set up an autoscaling group behind a load balancer. What I am not really understanding, however, is where my mongodb instance should live. I know that using DynamoDB it can be fairly intuitive to set up to work with that, but since I am not, my question is this: Where and how should mongo be set up to work with my app? Should it be on the same ec2 instance with my app, and if so, how does that work with new instances starting and being terminated? Should I set up an instance dedicated only for mongo? In addition, to that question, how do I create snapshots and backups of my data?
This is a good document for installing MongoDB on EC2, and managing backups: https://docs.mongodb.org/ecosystem/platforms/amazon-ec2/
If you aren't comfortable doing all this yourself you might want to also look into MongoLab which is a MongoDB as a Service that can run on AWS.
Your database should definitely be in a separate instance than your app, from all aspects.
A very basic tier based application should comprise of the app server cluster in a scaling group behind a load balancer - in a public subnet, and a separate cluster (recommended in a different subnet which is not publicly accessible), which your app cluster will speak to. whether to use an ELB for Mongo or not actually depends on your mongo config (replica set).
In regards to snapshots (assume this will only be relevant for your DB), have a look at this.
You can easily install MongoDB in AWS Cloud 9 by using the below process
First create Cloud 9 environment in AWS then at the terminal
ubuntu:~/environment $ At the terminal you’ll see this.
Enter touch mongodb-org-3.6.repo into the terminal
Now open the mongodb-org-3.6.repo file in your code editor (select it from the left-hand file menu) and paste the following into it then save the file:
[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
* Now run the following in your terminal:
sudo mv mongodb-org-3.6.repo /etc/yum.repos.d
sudo yum install -y mongodb-org
If the second code does not work try:
sudo apt install mongodb-clients
Close the mongodb-org-3.6.repo file and press Close tab when prompted
Change directories back into root ~ by entering cd into the terminal then enter the following commands:
“ubuntu:~ $ “ - Terminal should look like this.
sudo mkdir -p /data/db
echo 'mongod --dbpath=data --nojournal' > mongod
chmod a+x mongod
Now test mongod with ./mongod
Remember, you must first enter cd to change directories into root ~ before running ./mongod
Don't forget to shut down ./mongod with ctrl + c each time you're done working
-if this error pops up while using command mongod
exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/db, terminating
Then use the code:
sudo chmod -R go+w /data/db
Reference
Related
I have a few quick navigation plugins such as "block travel" I use all the time. Is there a way to use these in cloud shell?
I imagine there are some restrictions, but even some simple editor plugins can be huge timesavers.
While I'm at it - alt-D to duplicate a line, or transpose lines - some of those seem to be missing and hard to use key remapping to get working, at least within the shell. In general maybe keyboard shortcuts seem to get trapped by the browser or PWA wrapper. I'm using cloudshell as a webapp on a chromebook FWIW, for various secure projects.
I have come up with a solution that covers both aspects of your question
To get Unlimited Persistent Disk:
You can use Google Cloud Storage FUSE
Google Cloud Storage FUSE lets you mount a GCS bucket as a folder to your linux instance. By doing that you get an “unlimited '' persistent disk and it is super simple to set up since gcsfuse is already installed in cloud shell.
1. Create a GCS bucket (you only need to run this once) -- replace BUCKET_NAME with any name:
gsutil mb "gs://BUCKET_NAME/"
2. Create a local directory for mounting -- replace FOLDER_NAME with the chosen directory name:
mkdir /home/[USER]/[FOLDER_NAME]
chmod 777 /home/[USER]/[FOLDER_NAME]
3. Mount the bucket onto the local filesystem (note: you need to re-run this every time Cloud Shell starts)
gcsfuse -o nonempty -file-mode=777 -dir-mode=777 --uid=1000 --debug_gcs [BUCKET_NAME] /home/[USER]/[FOLDER_NAME]
To use third party plugins in cloud shell:
You can use an environment customization script (.customization_environment) as mentioned in the public documentation. It allows you to install additional packages into your Cloud Shell environment when it starts.
For reference, below are the steps to install VS Code plug-in.
Step 1:
To install the VSCode server, run the script named visual_studio_code.sh as below, in the root directory workspace of Cloud Shell Editor.
visual_studio_code.sh file:
export VERSION=`curl -s https://api.github.com/repos/cdr/code-server/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")'`
wget https://github.com/cdr/code-server/releases/download/$VERSION/code-server-3.10.2-linux-amd64.tar.gz
tar -xvzf code-server-3.10.2-linux-amd64.tar.gz
cd code-server-3.10.2-linux-amd64
Run the script using the below command in shell,
./visual_studio_code.sh
if getting permission denied error then run this following command in shell,
chmod +x visual_studio_code.sh
./visual_studio_code.sh
Step 2:
Make a customization script in the root directory workspace in Cloud Shell Editor to start VS Code Server on boot with the below commands :
.customization_environment file :
#!/bin/sh
#.customize_environmnet run in background as root, wait for your user to initialize
sleep 20
sudo -u [USER] /home/[USER]/code-server-3.10.2-linux-amd64/code-server --auth none --port 9090
Step 3:
To view Visual Studio Code server on port 9090 :
Click on Web Preview > Change Port > 9090
If getting a 404 error, remove ‘?authuser=0’ from the url.
Visual Studio Code Server will now be running on the browser!!!
Block travel navigation plugin:
To have the block travel navigation plugin in cloud shell,follow the below commands and run them in shell in root directory:
wget https://github.com/efatsi/block-travel/archive/refs/tags/v1.0.0.tar.gz
tar xzvf v1.0.0.tar.gz
ls
#You will see block-travel-1.0.0
block-travel-1.0.0/keymaps/block-travel.cson --auth none --port 9090
#You might get Permission denied if yes, then follow the next two commands else go to webport view in 9090
chmod +x block-travel-1.0.0/keymaps/block-travel.cson
block-travel-1.0.0/keymaps/block-travel.cson --auth none --port 9090
Open the webport view in 9090, you will be able to navigate through the vs code files using :
Alt+up for block-travel.jumpUp
Alt+shift+up for block-travel.selectUp
Alt+down for block-travel.jumpDown
Alt+shift+down for block-travel.selectDown
WARNING: This should not be considered a long term solution, just a stop gap until this is supported in an easier fashion.
This might not be the greatest idea but it does seem to work for the vim extension I tried in my environment. Probably best to make a request through the in product feedback to get it officially added but until then you can follow these steps.
Upload the .vsix package to your $HOME directory.
Unzip the package into the /google/devshell/editor/theia/plugins directory. This action will not persist so you'll want to add the command to the .customize_environment script actions.
e.g.
sudo unzip vscodevim.vsix -d /google/devshell/editor/theia/plugins/vscode-vim
Now for the questionable part. You'll want to install the pslist package to make life easy so you have access to the rkill command. You probably also want to add this to the .customize_environment file as well since it also will not persist.
sudo apt install pslist
Now we need to get the process id for the editor. Currently this seems to be spawned by a supervisord command, which also spawns the tmux section so we're going to grab the process id that is from the runuser command it spawns (and filter for the theia one just in case).
ps ax | grep runuser | grep "theia start"
Then we can use rkill to kill the process and all of the its children, which will cause supervisord to restart it for us and the plugin should be available.
sudo rkill PID_OF_GREP_OUTPUT
I'm not sure the best way to script the rkill command yet, since I don't know the timing of when it's up vs the .customize_environment execution, so right now I run this each time I start up a new VM.
If anything goes horribly wrong, you should be able to request a restart of the VM from the menu options and get a fresh one.
Cloud Shell offers VS Code editor experience through Theia. Did you try cloud Code editor in the cloud shell? that is exposed through "Open Editor" button on the top right, this will open cloud code editor that gives you VSCode experience. You have all the navigation keys that are available in the editor.
I recently installed docker and ran a container with mongodb, now my local brew's mongodb isn't working.
I am using a mac and originally installed mongodb using homebrew. brew services list now shows the mongodb service as 'running' but in yellow (apparently this means unknown), and I cannot connect using mongochef. This began happening after only once stopping the service brew service stop mongodb, as it was preventing me from using docker with this project: https://meanjs.org/. I stopped the service because there was a conflict for the mongodb port. Now even after destroying all the docker containers, I can't get mongodb working on my machine. I have tried reinstalling mongodb using brew, and stopping and starting the service.
I have also tried deleting the file in /tmp as here:
MongoDB Failing to Start - ***aborting after fassert() failure
Any suggestions about where to look and what to do?
So this turned out to be one of those really annoying problems that comes up when you have a lot of other stuff to do. I am still not completely sure if it was one issue or multiple issues. If I was to try and fix it again though, this is the order I would go in:
I removed mongo completely and reinstalled (this finally fixed the mongodb service's status from being shown in yellow instead of in green), as here: https://gist.github.com/katychuang/10439243:
# checks to see if running
launchctl list | grep mongo
launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.mongodb.plist
launchctl remove homebrew.mxcl.mongodb
pkill -f mongod
rm -f ~/Library/LaunchAgents/homebrew.mxcl.mongodb.plist
brew uninstall mongodb
# double check existence
ls -al /usr/local/bin/mong*
ls -al ~/Library/LaunchAgents
# this was mentioned in the comments
rm -R /usr/local/var/mongodb
However, I think my data was corrupted too. Unfortunately mongod --repair (mongodb error: how do I make sure that your journal directory is mounted) was saying it couldn't repair the data without running with --journaled but that didn't work either. So I deleted my databases in /data/db I didn't need them anyway because I have backups and just restored them easily.
Hope that can help anyone having similar issues.
Sidenote:
The mongod command on the command line gave some helpful output in tracking this down.
Also vim ~/Library/LaunchAgents/homebrew.mxcl.mongodb.plist allowed me to see which command was being run:
<string>/usr/local/opt/mongodb/bin/mongod</string>
<string>--config</string>
<string>/usr/local/etc/mongod.conf</string>
This showed me that /usr/local/etc/mongod.conf was being used, and realising that whilst running /usr/local/opt/mongodb/bin/mongod was working, running:
/usr/local/opt/mongodb/bin/mongod --config /usr/local/etc/mongod.conf was not working, helped me narrow the problem down
I'm getting the following error when I try to run "mongod" in the terminal. I've tried uninstalling, reinstalling, and restarting the machine. Any suggestions on how to get it working would be amazing.
ERROR:
dbpath (/data/db) does not exist.
Create this directory or give existing directory in --dbpath.
See http://dochub.mongodb.org/core/startingandstoppingmongo
Side note:
Node also stopped working on my machine around the same time that I got this error.
events.js:72
throw er; // Unhandled 'error' event
^
Error: failed to connect to [localhost:27017]
Any help would be much appreciated!
This should work to ensure that the directory is set up in the right place so that Mongo can find it:
sudo mkdir -p /data/db/
sudo chown `id -u` /data/db
You need to create the directory on root /data/db or set any other path with the following command :
mongod --dbpath /srv/mongodb/
See the example link
I solved the problem with :
sudo mongod --dbpath=/var/lib/mongodb and then mongo to access the mongodb Shell.
Change the user of the new data directory:
chown mongodb [rute_directory]
And try another time to start the mongo service
service mongod start
I solve the same problem with this.
Daemons (usually ending with d) are normally started as services. Starting the service (daemon) will allow mongodb to work as designed (without permission changes if integrates well with your distro). I start it using the service named mongodb instead of starting mongod directly--on distro with systemd enable on startup then run like:
sudo systemctl enable mongodb
sudo systemctl start mongodb
or, on distro with upstart (if you have /etc/init) or init (if you have /etc/init.d) ( https://www.tecmint.com/systemd-replaces-init-in-linux/ ) instead run:
sudo service mongodb enable
sudo service mongodb start
If you have a distro with rc ("run commands") such as Gentoo (settings in /etc/init.d) (https://forums.gentoo.org/viewtopic-t-854138-start-0.html) run:
rc-update add mongodb default
/etc/init.d/mongodb start
In a distro/version of FreeBSD which still has rc (check whether your version switched to systemd, otherwise see below):
add the following line to /etc/rc.conf:
mongod_enable="YES"
then:
sudo service mongod start
After starting the service, an unpriveleged user can use mongo, and each user will have separate data.
I also got the error that "The file /data/db doesn't exist" when I tried to save my file using the "mkdir -p /data/db" command(using both with and without sudo command). But later on one site, a person named Emil answered that the path "/data/db" no longer works on Mac, so use "~/data/db" instead
i.e., use the command
mkdir -p ~/data/db
instead of previous command.
Moreover, use
mongod --dbpath ~/data/db
to run mongod
It worked for me, hope it work for others too facing the same problem
Ever since I changed the dbpath in /etc/mongodb.conf, MongoDB has not been starting automatically, nor using the new dbpath. Prior to the change, MongoDB would be running when the computer started and I was able to simply run the command mongo to get into the console or start my Ruby on Rails server with no issues.
After I made the modification (in order to switch to a new drive with more space), the only way I can get everything to work is by manually running the command mongod --config /etc/mongodb.conf. If I don't run that, it doesn't seem like the service is running and running without the --config option give me the following error: ERROR: dbpath (/data/db/) does not exist. even though the config file says nothing about data/db.
Some other notes:
In addition to changing /etc/mongodb.conf, I moved all files out of /var/lib/mongodb and into /home/nick/appdev/mongodb.
I changed the owner and group from root to nick. Tried changing it back, but it didn't seem to fix anything.
I'm running Ubuntu 12.10 Beta 1 and Mongo 2.2.0 with Ruby on Rails 3.2.8
A late follow up on the above question...
I had a similar issue after moving the db to an ebs on ec2.
It turns out that just running mongod still directs the dbpath to /data/db/ (which exists).
The /etc/mongodb.conf is completely ignored unless specifically directed to.
I manage to work around this by using the directive --config or just the --dbpath(both work)
But was left wondering where does mongod takes it defaults from...?!
I was unable to locate and override these defaults anywhere.
Anyone ?
Note:
I am really annoyed by this behaviour of mongod...This is just bad design,and bad documentation.
It turns out that I needed to set the owner and group to mongodb. When I transferred the files to the new directory, I had set the owner and group to my user account nick and also tried root, neither of which worked.
To do so, here are the following commands:
sudo chown mongodb /home/nick/appdev/mongodb -R
sudo chgrp mongodb /home/nick/appdev/mongodb -R
To confirm that it worked, you can check the file permissions with:
ls -l /home/nick/appdev/mongodb
After checking all permission in the data, journal and log folders as suggested, my problem was solved by giving permission to a lock file in the /tmp folder
sudo chown mongod:mongod mongodb-27017.sock
I was running it as a AWS Amazon Linux instance. I figured that out by executing as the mongod user as below, and then, researching the error code. It might be useful for other troubleshooting.
sudo -S -u mongod mongod -f /etc/mongod.conf
MongoDB 1.6 is very old and the latest production version is 2.2, which contains a large amount of bug fixes and enhancements since 1.6.
Am I correct that you haven't installed 1.6 via a package manager such as yum or aptitude? I don't believe there are packages for 1.6 at present afaik. Therefore, mongod is behaving correctly as you have not started MongoDB with a control script.
Please see this link on configuration file options.
I am in the process of moving my mongo data to a different drive. All of the data I want to move is stored in /data/db and I am moving it to a NAS (Network attached storage).
First step:
mongodump -d mydb -c mycollection -o nas/mongo-temp
This created a file tree in mongo-temp/ like so:
dump
`-- mydb
`-- mycollection.bson
1 directory, 1 file
I then stopped the mongod service and created a new /data/db directory:
/etc/init.d/mongod stop
mkdir mongo-temp/data/db
...and changed the dbpath line in /etc/mongodb.conf
dbpath=.../mongo-temp/data/db
I successfully restarted the mongo server using /etc/init.d/mongod start.
When I try to connect:
mongo
MongoDB shell version: 1.6.4
Thu May 3 09:53:23 *** warning: spider monkey build without utf8 support. consider rebuilding with utf8 support
connecting to: test
Thu May 3 09:53:24 Error: couldn't connect to server 127.0.0.1 (anon):1154
exception: connect failed
I've tried to start mongod with the command mongod --dbpath .../mongo-temp/data/db but I get an error that says:
Thu May 3 09:57:26 exception in initAndListen std::exception: Unable to acquire lock for lockfilepath: /home/dlpstats/nas-mnt/mongo-temp/data/db/mongod.lock
Removing the lockfile doesn't help. If I run the mongod command without --dbpath, the server starts fine and I am able to make queries on my old database.
First, you mentioned that you used mongodump to populate the new drive - was this just a method of backing things up or did you intend that to be the new database files? That is not how it works - mongodump output is not the same as a database file - it needs to be re-imported with mongoresore in fact. If you do a straight data file copy then the transfer will be seamless.
Then, as well as the permissions suggested by Wes in his answer, a few more things to check:
That you have shut down the old server successfully and completely - it's possible it's mis-reported error and you are getting it because it is trying to grab a port that is already open
You are using version 1.6.4 according to the mongo shell output, my guess is that you installed from the Ubuntu repo for 11.04 or similar, that is not a good option - 1.6 is very old at this point. Use the 10gen repos (http://www.mongodb.org/display/DOCS/Ubuntu+and+Debian+packages) or download the binaries and get a more recent version
Last but not least, when you start the mongod manually, make sure all the arguments are the same, like the port. When you connect via the mongo shell, specify the port you started the mongod on - don't rely on defaults when running into issues like this, be explicit.
I faced this problem and issuing following command solved my problem:
rm /var/lib/mongodb/mongod.lock
And then restart the mongod.
But I'm not sure is it a good solution or not.
Check the permissions for the directory and parent directories of mongo-temp. Presumably it's running as the mongodb user?
You need execute permissions on the directory (and parent directories) in order to create files there. Execute permissions on a directory allow you to list the files there, which is needed to be able to open the file for writing.