How can I conect a NodeJS app to a MongoDB running in a Docker container on AWS? - mongodb

I am attempting to deploy my first MEAN stack application ('weatherapp') to production on AWS.
I deployed my NodeJS/Express/Angular app to AWS Elastic Beanstalk (preconfigured Linux machine running Node). This works fine and I can view the app in the browser.
Separately I created a docker container running MongoDB and deployed it to AWS / EC2 following the steps in this post:
https://blog.codeship.com/running-mean-web-application-docker-containers-aws/
My question is - how do I connect the two?
In my NodeJS app I was connecting to my local Mongo instance locally like this:
'mongodb://localhost:27017/weatherapp'
What steps can I take to find out what the connection string should be for my production Mongo instance on docker?
Thanks in advance!

The answer to this is two-fold. We need to set some options on the Docker side in the EC2 instance and then some security groups and configuration on the AWS side. First, we'll start on on the Docker container side.
Container
When you run the MongoDB container, you will want to do two things:
Persist the data to disk.
Open the MongoDB port to the container.
To persist the data to disk you will want to do something like -v /data/db:/data/db. This will make the MongoDB data available at /data/db on the host. This makes sure that an accidental deletion or upgrade of the container doesn't lose any data.
Next, we need to publish the MongoDB port so that applications external of Docker can connect to it. The default MongoDB port is 27017 so let's publish that using -p 27017:27017.
If your original command for starting MongoDB was:
docker run --name mymongodb -d mongo
Then the new one would be:
docker run --name mymongodb -d -p 27017:27017 -v /data/db:/data/db
AWS
Now, we need to edit the security group of your EC2 instance and configuration of Elastic Beanstalk.
Security Groups
First, take a look at your Security Groups in the EC2 console. You will have a group for the Elastic Beanstalk application named similar to awseb-e-xanf9hqrw3-stack-AWSEBSecurityGroup-1N2T1AI2H05I8 with a ID similar to sg-07fb8c43. We'll use this ID in the next step so copy it somewhere.
Now find the Security Group attached to your EC2 instance running the Docker container. You will need to add a new rule to this group allowing access to the MongoDB container. Edit the group and add a new inbound rule for:
Type: Custom TCP
Protocol TCP
Port range: 27017
Source: sg-07fb8c43
This will allow the Elastic Beanstalk EC2 instances (using sg-07fb8c43) to access the MongoDB port on your Docker EC2 instance.
Elastic IP
You'll likely want a more static IP address for your EC2 instance in case it reboots. Navigate to the Elastic IPs section of the EC2 console and allocate a new address to your Docker EC2 instance.
The new Elastic IP will be the address you use in your Elastic Beanstalk configuration to connect to MongoDB. If your address was 54.67.29.50 then your application would connect to mongodb://54.67.29.50:27017.
Elastic Beanstalk
Now, instead of hardcoded this address in your Node.js application, you should configure your application to pull the information from an environment variable. In your application, you should read the MongoDB URL from something like process.env.MONGO_URI. Then, in your Elastic Beanstalk application configuration, navigate to the Software Configuration and then down to Environment Properties. Here, you create a property name of MONGO_URI and the value as mongodb://54.67.29.50:27017. This will allow you to easily change the MongoDB instance should it ever change or if you launch multiple environments with different databases.

Related

Connect PostgreSQL to rabbitMQ

I'm trying to get RabbitMQ to monitor a postgresql database to create a message queue when database rows are updated. The eventual plan is to feed this message queue into an AWS EKS (Elastic Kubernetes Service) cluster as a job.
I've read many many approaches to this but they are still confusing as a newcomer to RabbitMQ and many seemed to be written more than 5 years ago so I'm not sure if they'll still work with current versions of postgres and rabbitmq.
I've followed this guide about installing the area51/notify-rabbit docker container which can connect the two via a node app, but when I ran the docker container it immediately stopped and didn't seem to do anything.
There is also this guide, which uses a go app to connect the two, but I'd rather not use Go ouside of a docker container.
Additionally, there is also this method, to install the pg_amqp extension from a repository which hasn't been updated in years, which allows for a direct connection from PostgreSQL to RabbitMQ. However, when I followed this and attempted to install pg_amqp on my Postgres db (postgresql 12), I was unable to connect using psql to the database, getting the classic error:
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
My current set-up, is I have a rabbitMQ server installed in a docker container in an AWS EC2 instance which I can access via the internet. I ran the following to install and run it:
docker pull rabbitmq:3-management
docker run --rm -p 15672:15672 -p 5672:5672 rabbitmq:3-management
The postgresql database is running on a separate EC2 instance and both instances have the required ports open for accessing data from each server.
I have also looked into using Amazon SQS as well for this, but it didn't seem to have any info on linking Postgresql up to it. I haven't really seen any guides or Stack Overflow questions on this since 2017/18 so I'm wondering if this is still the best way to create a message broker for a kubernetes system? Any help/pointers on this much appreciated.
In the end, I decided the best thing to do was create some simple Python scripts to do the LISTEN/NOTIFY steps and route traffic from PostgreSQL to RabbitMQ based off the following code https://gist.github.com/kissgyorgy/beccba1291de962702ea9c237a900c79
I set it up inside Docker containers and set them to run in my Kubernetes cluster so they are within the automatic restarts if they fail.

Run a K3S server in a docker container, and connect a K3S agent in another docker container

I know k3d can do this magically via k3d cluster create myname --token MYTOKEN --agents 1, but I am trying to figure out how to do the most simple version of that 'manually'. I want to create a server something like:
docker run -e K3S_TOKEN=MYTOKEN rancher/k3s:latest server
And connect an agent something like like:
docker run -e K3S_TOKEN=MYTOKEN -e K3S_URL=https://localhost:6443 rancher/k3s:latest agent
Does anyone know what ports need to be forwarded here? How can I set this up? Nearly everything I try, the agent complains about port 6444 already in use, even if I disable as much as possible about the server with any combination of --no-deploy servicelb --disable-agent --no-deploy traefik
Feel free to disable literally everything other than the server and the agent, I'm trying to make this ultra ultra simple, but just butting my head against a wall at the moment. Thanks!
The containers must "see" each other. Docker isolates the networks by default, so "localhost" in your agent container is the agent container itself.
Possible solutions:
Run both containers without network isolation using --net=host, map API port of the server to the host with --port and use the host IP in the agent container or use docker-compose.
A working example for docker-compose is described here: https://www.trion.de/news/2019/08/28/kubernetes-in-docker-mit-k3s.html

copying mongodb data from VM to AWS instance

I have mongodb running on one of the VM internally. Now, we are moving the service on AWS.
How should I transport the mongodb data from VM to AWS instance? Mongo is running inside docker container in AWS.
Should I use MongoDump and MongoRestore? or any other approach here?
Also, I don't see mongod running as a service on AWS instance, since they are running inside the docker container. So, should I need to install the mongodb package and then do mongorestore?
Any help or thoughts here?

IBM Bluemix : Bulk load data into MongoDB

I have created a MongoDB service in Bluemix and I can successfully access it in an app deployed on Bluemix. I can create data in the MongoDB instance programmatically through my app, but what I I want to do is load data into MongoDB from my laptop.
I am not able to ping the MongoDB web address from my laptop, so I can not connect it from a standalone java program.
What is the way ahead to bulk load data into MongoDB on Bluemix ?
You can not connect to this experimental service from outside of Bluemix.
mongodb: You can not connect to this experimental service from outside of Bluemix. If you want to use your standalone java program to interact with this service on Bluemix, consider pushing your standalone java program as another application to Bluemix.
cf push mystandaloneapp -p standalone.jar --no-route
Then, bind the same mongodb instance to this application. When you restage the application, it should get the credentials in the VCAP_SERVICES environment variable.
mongolab: Assuming you created the mongolab service, from your Bluemix Dashboard, find and click on your MongoLab instance. From there, launch the MongoLab Dashboard. Click on your deployment (IbmCloud_***). You should see instructions on how to connect to mongo from shell, as well as import/export commands.
mongoimport -h ds049570.mongolab.com:49570 -d IbmCloud_ee4rm8hq_ecl23uf8 -c <collection> -u <user> -p <password> --file <input file>
You should also be able to connect to this from your java program.
Finally, check out the MongoDB by Compose service, which is an IBM provided MongoDB service, with a dashboard.

How can I connect a docker container in AWS to a RDS

How can I --link a docker container(odoo) running on a plain EC2 instance or via EB to a RDS in AWS? I tried loading a custom config file with the location of the server but that didn't work. In EB I created an app with a postgres DB, successfully deployed my docker.aws.json but I can not connect to web interface of the the application.
When I check the docker logs of the container it says everything started fine but expects the DB on localhost.
So like I said my question is how can I tell a docker container to --link to a RDS and not an other docker container à la --link db:db?
If you created a database in AWS Elastic Beanstalk, you can access it using environment variables that EB set for you. See examples in the documentation: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create-deploy-python-rds.html (Python example)
The environment variable containing the RDS DB hostname is called RDS_HOSTNAME. See example: https://github.com/awslabs/eb-demo-php-simple-app/tree/docker-apache
It's not possible to use a link as your RDS DB is not a Docker container.