I'm using MongoDB change stream in order to have indirect communication between servers.
One API server is in DMZ and the other is in Intranet, DB server is also in DMZ and both API servers are allowed to communicate with DB via 27017 port.
DMZ API server is doing inserts to DB and is listening for "update" event in order to return response to user, while Intranet API server is listening for insert event and doing updates to those documents only. Once Intranet API updates the document, response is returned to user from DMZ API. Hope this makes sense so far.
That being a setup, I have an issue with DB server. It's complaining about swap memory being full always and since it's replica set, there are 3 servers in it and each has 4GB of RAM.
Do I need to add more RAM, and how much if anyone knows?
Related
I'm currently using Heroku auto scale for my servers. And I need to setup a scalable app using socket.io to allow instant updates of data (bear in mind that it's only for updating the frontend displays, no data is processed).
The way I was going to set it up was as follows:
In the image, all the servers have a socket connection to the "main" socket.io server and to the user.
A user would do an action through an API, the server would do its' thing (save to mondoDB or compute...) and pass it to a "main" socket.io server through its socket.io connection but would not send anything back to the user. The "main" server would receive the request through the socket.io connection and emit it back to the servers, which would then emit it to their users.
So the flow would be: User > Server > Main socket.io server > Server > User
My questions are:
Would this work?
Why do all the docs refer to db type redis?
I am still new to DigitalOcean and Ubuntu servers, so I hope one of you can explain to me the difference between Inbound and Outbound traffic. I tried reading through DigitalOcean's information, but it still doesn't inform exactly what i can count as inbound and outbound.
As far as i can see, it is only the outbound traffic that is being billed. So my question is, what is counted as outbound traffic?
Is it outbound traffic when i log into my server and run "npm install" while creating docker containers, or is that inbound?
Is it outbound or inbound when i run "git clone"?
I hope one of you can give me an explanation of what is categorised as inbound and outbound.
First off, you're highly unlikely to hit your bandwidth limit on DigitalOcean even if you are on the lowest of the low end droplet with no other bandwidth-boosting services.
That said, there are four types of traffic on DigitalOcean when it comes to droplets: outbound local, inbound local, outbound remote, and inbound remote. Any transaction will do some of both, but likely will be skewed in the direction of one or the other.
Case in point: I have a droplet that runs a Neo4j database server, and another droplet that queries that server with Neo4j (graph database) drivers. When my client server makes a request to the database, it sends a database query through the driver, which is a small outbound remote from the client POV (the servers are in separate projects, so it's not "internal traffic" like if they were using internal IPs in the same project) and a small inbound remote for the database server. The actual substantive data transfer is the query response, which in the case of my workloads tends to be a few hundred MB at minimum, which shows up as remote outbound for the database server and remote inbound for the client.
To sum up: any transfer outside of DigitalOcean resources in the same virtual network counts as outgoing when you send data (ev like curl or wget) and a likely much larger inbound when you receive it. Vice versa if you're responding to requests with a lot of data.
I have 2 x AWS EC2 instances with a node.js app. Out of the box, they come with a local mongod instance that works fine. Given the criticality of the app, I decided to spin up 2 x EC2 front ends (node js) to talk to a mongo db in another availability zone using the AWS ELB.
Full IP communication/27017 connectivity exists between all 3 nodes.
When using only 1 server to the mongo server, it works just fine. When adding both front end servers into the ELB target group, I get random 504 gateway errors.
Removing a server from the group fixes the issue.
Any suggestions on what I should look for?
In terms of how the node.js server connects to mongo, there is a config.json file that simply points out the IP and DB name required.
Thanks!
AWS Load Balancer use "round robin" mechanism to route user's requests. Does your application have way to control user sessions? If not then your first request come to server 1, then second request to server 2 which doesn't have any information to the first request may result in error. That explain why it works fine when you have 1 server only
the server uses redis (the app server) and the following components:
Node.js - Server-side Javascript-framework
Express.js - Web application framework for Node.js
Nginx - Web server & reverse proxy
MongoDB - NoSQL database
redis - Session Manager & data structure server
Socket.IO - Bi-directional communication between web clients and servers
I am setting up two computers to run a web application. web-host hosts a MongoDB database and NodeJS web server, while worker runs some more demanding processes and populates the database. Using an SSH tunnel from worker, web-host:27017 is accessible using localhost:9999 from worker. web-host:80 has been set up to be accessible on http://our.corporate.site/my_site/.
At the moment MongoDB has no authentication on it - anything that can contact web-host:27017 can read or write anything to the database.
With this setup, how paranoid should I be about authenticating requests to MongoDB? The answers to this question seemed to suggest not very. Considering access is only possible from localhost it seems about as secure as the local file system. In MySQL I usually have a special 'web' user with limited privileges to limit the damage of an injection attack in case I make a mistake sanitizing input, however MongoDB seems less vulnerable to injection (or at least easier to sanitize) compared with MySQL.
Here's the issue: If you do set-up Mongo authentication, you are going to need to store the keys on the machine that accesses it.
So assuming that web-host:80 is compromised, the keys are also vulnerable.
There are some mitigation processes you can use to secure your environment, but there is no silver bullet if an attacker gains root access to your environment.
First I would consider putting mongodb on a separate machine on a private internal network that can only be accessed by machines in a DMZ (the part of the network where machines can communicate with your internal network and the outside world).
Next, assuming you are running a Linux-based system, you should be able to use AppArmor or SELinux to limit which processes are allowed to make outbound network requests. In this case only your webapp process should be able to initiate network requests such as connecting to your Mongo database.
If an attacker was able to get non-root access on your machine, the SELinux/AppArmor system policy would prevent them from initiating a connection to your database from their own script.
Using this architecture, you should be more secure than simply augmenting your current architecture with authentication. In a choice between the SELinux/AppArmor, I would use SELinux, since it is was much more mature and had much more granular control the last time I checked.
My question, to be more clear, it is to create a server with mongodb on a cloud hosting (for example) and access it through another server.
Example:
I have a mobile app.
I hosted my mongoDB a cloud hosting (ubuntu).
I want to connect my app to the db on the server cloud.
Is it possible? How?
I'm joining this learning and my question was exactly MongoDB to create a server in a way that I could access it remotely.
Out of "localhost"? Different from all the tutorials I've seen.
From what you are describing, I think you want to implement a 2-Tier-Architecture. For practically all use cases, don't do it!
It's definitely possible, yes. You can open up the MongoDB port in your firewall. Let's say your computer has a fixed IP or a fixed name like mymongo.example.com. You can then connect to mongodb://mymongo.example.com:27017 (if you use the default port). But beware:
Security You need to make sure that clients can only perform those operations that you want to allow, e.g. using MongoDB integrated authentication, otherwise some random script kiddie will steal you database, delete it, or fill it with random data. Many servers, even if they don't host a well-known service, get attacked thousands of times per day. Also, you probably want to encrypt the connection so people can't spy on the connection. And to make it all worse, you will have to store the database credentials in your client app, which is practically impossible to do in a truly secure way.
Software architecture There is a ton of arguments against this architecture, but 1) alone should be enough. You never want to couple your client to the database, be it because of data migrations, software updates, security considerations, etc.
3-Tier
So what to do instead? Use a 3-Tier-Architecture: Host a server of some kind on mymongo.example.com that then connects to the database. That server could be implemented in nginx/node.js, iis/asp.net, apache/php, or whatever. It could even be a plain old C application (like many game servers).
The mongodb can still reside on yet a different machine, but when you use a server, the database credentials are only known to the server, not to all the clients.
Yes, it is possible. You would connect to MongoDB using the ip address of your host, or preferably using it's fully qualified hostname rather than "localhost". If you do that, you should secure your MongoDB installation otherwise anyone would be able to connect to your MongoDB instance. At an absolute minimum, enable MongoDB authentication. You should read up on MongoDB Security.
For a mobile application, you would probably have some sort of application server in front of MongoDB, e.g. your mobile application would not be connecting to MongoDB directly. In that case only your application server would be connecting to MongoDB, and you would secure MongoDB accordingly.