I want to run a socket program in aws ecs with client and server in one task definition. I am able to run it when I use awsvpc network mode and connect to server on localhost every time. This is good so I don’t need to know the IP address of server. The issue is server has to start on some port and if I run 10 of these tasks only 3 tasks(= number of running instances) run at a time. This is clearly because 10 tasks cannot open the same port. I can manually check for open ports before starting the server and somehow write it to docker shared volume where client can read and connect. But this seems complicated and my server has unnecessary code. For the Services there is dynamic port mapping by using Application Load Balancer but there isn’t anything for simply running tasks.
How can I run multiple socket programs without having to manage the port number in Aws ecs?
If you're using awsvpc mode, each task will get its own eni and there shouldn't be any port conflict. But each instance type has a limited number of enis available. You can increase that by enabling eni trunking which, however is supported by a handful of instance types:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/container-instance-eni.html#eni-trunking-supported-instance-types
Related
On Google Cloud Platform I need to create two virtual machines that will act as the main server and replication server (as a database).
It happens that I will have several applications that will connect to the main server, which requires me to define in these applications the local network IP (VPC) of this main machine.
My problem occurs when there is some failure on the main machine or even an emergency/maintenance reboot. This type of operation will require me to urgently change all applications to use the replication machine's VPC IP instead of the main one.
Is there any way I can have one IP that can be dedicated to connect to the main machine, but when necessary, change its destination to be the replication machine?
Instead use an internal L7 load balancer. See the comparision in order to decide if this is suitable. This PDF explains the stack - and envoyproxy.io is the load balancer.
Andromeda even implements round robin, but for NIC instead of IP.
Also see: Patterns for using floating IP addresses in Compute Engine
I have am trying to set up Data Factory between the on premise SQL Server on a corporate network and a SQL Server hosted in an Azure VM (not MI).
Is this possible? I set up both nodes on the IR and opened firewall port 8060 both on premise and in the Azure VM.
My goal is to copy data as needed from the on premise sql server to the sql server hosted in the azure vm.
I am getting this error
This node has some connectivity issue with the dispatcher node. Please check the connectivity between the nodes within your network.
The Integration Runtime (Self-hosted) node is trying to sync the credentials across nodes. It may take several minutes.
If this warning appears for over 10 minutes, please check the connectivity with Dispatcher node.
In my case port 8060 is being blocked by the corporate firewall, not the server firewall. Since I can't change the corporate firewall, I'll use a different work around. However I can't find a good demo of this working in two separate networks. All the demos use Azure VM's, so I was hoping to test out a real life example.
I deployed a mongodb in the default docker network bridge.
Please recall that, the gateway of the bridge network is 172.17.0.1.
For more information, refer to https://docs.docker.com/network/network-tutorial-standalone/.
Recently, I discovered that the mongodb receives a lot of slow queries from a process running behind 172.17.0.1:39694
How do I find out what process is running on the gateway port 172.17.0.1:39694?
docker network inspect bridge
shows only nodes within the bridge network, but shows nothing related what processes are running on its gateway ports.
Each MongoDB client identifies itself when it establishes the connection. Example:
{"t":{"$date":"2020-11-25T10:49:02.505-05:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn216","msg":"client metadata","attr":{"remote":"127.0.0.1:58122","client":"conn216","doc":{"driver":{"name":"mongo-ruby-driver","version":"2.14.0.rc1"},"os":{"type":"linux","name":"linux-gnu","architecture":"x86_64"},"platform":"Ruby 2.7.1, x86_64-linux, x86_64-pc-linux-gnu"}}}
This gives you the language, driver and driver's version.
You can pass additional metadata to identify connections. For example in Ruby you would do this via Client#initialize :app_name option.
For mapping ports to processes, see e.g. https://www.putorius.net/process-listening-on-port.html
I am testing the reliability of TCP connections using Amazon Elastic Load Balancer compared to not using the Load Balancer to see if it has any impact.
I have setup a small Elastic Load Balancer on Amazon EC2 us-east zones with 8 t2.micro instances using an auto scaling group without policy and set to 8 min/max instance.
Each instance run a simple TCP server that accept connections on port 8017 and relay some data to the clients coming from another remote server located in my network. The same data is send to all clients.
For the purpose of the test, the servers running on the micro instances are only sending 1 byte of data every 60 seconds (to be sure the connection don't time out).
I connected multiple clients from various outside networks using the ELB DNS name provided, and after maybe 6-24 hours, I always stop receiving data and eventually the connections all die.
All clients stops around the same time, even though they are on different network/ISP. Each "client" application is doing about 10 TCP connections and they all stop receiving data.
All server instances look fine after this happen, they still send data.
To do further testing and eliminate the TCP server code problem, I also have external clients connected directly to the public IP of a single instance, without the ELB, and the data doesn't stop and the connection is not lost in this case (so far).
The Load balancer Idle Timeout is set to 900 seconds.
The Cross-Zone load balancing is enabled and I am using the following zones: us-east-1e, us-east-1b, us-east-1c, us-east-1d
I read the documentation, and searched everywhere to see if this is a known behaviour, but I couldn't find any clear answer or confirmation of others having the same issue, but it seems clear it is happening in my case.
My question: Is this a known/expected behaviour for TCP load balancer? Otherwise, any idea what could be the problem in my setup?
I have an environment (Graphite) that looks like the following:
N worker servers
1 relay server that forwards work to these worker servers
1 web server that can query the relay server.
I would like to use Chef to setup and deploy this environment in EC2 without having to create each worker server individually, get their IPs and set them as attributes in the relay cookbook, create that relay, get the IP, set it as an attribute in the web server cookbook, etc.
Is there a way using chef in which I can make sure that the environment is properly deployed, configured and running without having to set the IPs manually? Particularly, I would like to be able to add a worker server and have the relay update its worker list, or swap the relay server for another one and have the web server update its reference accordingly.
Perhaps this is not what Chef is intended for and is more for per-server configuration and deployment, if that is the case, what would be a technology that facilitates this?
Things you will need are:
knife-ec2 - This is used to start/stop Amazon EC2 instances.
chef-server - To be able to use search in your recipes. It should be also accessible from your EC2 instances.
search - with this you will be able to find among the nodes provisioned by chef, exactly the one you need using different queries.
I have lately written an article How to Run Dynamic Cloud Tests with 800 Tomcats, Amazon EC2, Jenkins and LiveRebel. It involves loadbalancer installation and loadbalancer must know all IP adresses of the servers it balances. You can check out the recipe of balanced node, how it looks for loadbalancer:
search(:node, "roles:lr-loadbalancer").first
And check out the loadbalancer recipe, how it looks for all the balanced nodes and updates the apache config file:
lr_nodes = search(:node, "role:lr-node")
template ::File.join( node[:apache2][:home], 'conf.d', 'httpd-proxy-balancer.conf' ) do
mode 0644
variables(:lr_nodes => lr_nodes)
notifies :restart, 'service[apache2]'
end
Perhaps you are looking for this?
http://www.infochimps.com/platform/ironfan