We are using Haproxy on top of Mesos cluster, We are doing dynamic reloads for Haproxy based on marathon events (50-100 times in a day). We have nearly 300 applications that are running on Mesos (300 virtual hosts in Haproxy). When we do dynamic reloads, Haproxy is taking long time to reload Haproxy, we observed that for 50 applications takes 30-40secs to reload Haproxy. We have a single config file for Haproxy, when we do reload all the applications are getting reloaded (Front-ends), this causing downtime of all applications. Is there anyway to reduce the downtime and impact on end-users.
We tried this scenario,
"http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html"
By this if user requests while reload, the requests are queued and serving after reload.
But if we do multiple reloads one after another, HaProxy's old processes persist even after reloading the HaProxy service, this is causing serious issues.
root 7816 0.1 0.0 20024 3028 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778
root 7817 0.0 0.0 20024 3148 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778
Is there any solution stop the previous process once after it serving the request.
Can we separate the configurations based on front-ends like in Nginx, so that only those apps will effect if there is any changes in backend.
Thank you.
We found the issue, it's because of DNS, Haproxy is taking much time for resolving the URLS, we added the /etc/hosts file for local cache DNS..
Now it's taking 30ms for a reload.
Thank you.
Related
I have a dedicated server running Ubuntu 20.04, with cPanel 106.11, MySQL 8, PHP 8.1, Elasticsearch 7.17.8 and i run magento 2.4.5-p1. Config Server Security & Firewall is enabled.
Every couple of days i get an monitoring alert to say my server doesnt respond to ping and the host has to do a hard reboot, they are getting frustrated with this and say they will turn off monitoring unless i sort this as they have checked all hardware which is fine.
This happens at different times and usually overnight.
I have looked through syslog, mysql log, elasticsearch log, magento 2 logs, apache log, kern.log and i cant find the cause of the issue.
I have enabled "sar" and the RAM usage around the time is 64%, cpu usage is between 5-10%.
What else can i look at to try and diagnose this issue?
Additional info requested by Wilson:
select count - https://justpaste.it/6zc95
show global status - https://justpaste.it/6vqvg
show global variables - https://justpaste.it/cb52m
full process list - https://justpaste.it/d41lt
status - https://justpaste.it/9ht1i
show engine innodb status - https://justpaste.it/a9uem
top -b -n 1 - https://justpaste.it/4zdbx
top -b -n 1 -H - https://justpaste.it/bqt57
ulimit -a - https://justpaste.it/5sjr4
iostat -xm 5 3 - https://justpaste.it/c37to
df -h, df -i, free -h and cat /proc/meminfo - https://justpaste.it/csmwh
htop - https://freeimage.host/i/HAKG0va
Server is using nvme drives, 32GB RAM, 6 cores, MySQL is running on same server as litespeed.
Server has not gone down again since posting this but the datacentre usually reboot within 15 - 20 mins and 99% of the time happens overnight. The server is not accessible over ssh when it crashes.
Rate Per Second = RPS
Suggestions to consider for your instance (should be available in your cpanel as they are all dynamic variables)
connect_timeout=30 # from 10 seconds to reduce aborted_connects RPHr of 75
innodb_io_capacity=900 # from 200 to use more of NVME IOPS capacity
thread_cache_size=36 # from 9 to reduce threads_created RPHr of 75
read_rnd_buffer_size=32768 # from 256K to reduce handler_read_rnd_next RPS of 5,805
read_buffer_size=524288 # from 128K to reduce handler_read_next RPS of 5,063
Many more opportunities exist to improve performance of your instance.
View profile for contact info, please. We are pushing the one question/one answer planned for this platform.
I have referred to https://redis.io/topics/mass-insert and tried the Luke protocol,
and did
cat data.txt | redis-cli -a <pass> -h <events-k8s-service> --pipe-timeout 100 > /dev/null
The redirection to /dev/null is to ignore the replies. The CLIENT REPLY of redis can't serve its purpose here from CLI and it may turn into a blocking command.
The data.txt has around 18 Million records/commands like
SELECT 1
SET key1 '"field1":"val1","field2":"val2","field3":"val3","field4":"val4","field5":val5,"field6":val6'
SET key2 '"field1":"val1","field2":"val2","field3":"val3","field4":"val4","field5":val5,"field6":val6'
.
.
.
This command is executed from a cronJob which execs into the redis pod, and executes the above command from within the pod, to understand the footprint, the redis pod had no resources limit and following are the observation:
Keys loaded: 18147292
Time taken: ~31 minutes
Peak CPU: 2063 m
Peak Memory: 4745 Mi
The resources consumed are way too high and the time taken is too long.
The questions:
How do we load mass load data of the order 50 Million keys using redis pipe, is there an alternate approach to this problem ?
Is there a golang/python library that does the same mass loading efficiently(less time , little footprint of mem and cpu) ?
Do we need to fine tune redis here ?
The help is appreciated, Thanks in advance.
If you are using the redis-cli inside the pod to move the millions of key into Redis POD won't be able to handle it.
Also, you have not specified any resources that you are giving to Redis however it's a memory store so it will be better to give proper memory to redis 2-3 GB depends on usegae.
You can try out the Redis-riot : https://github.com/redis-developer/riot
to add data into the Redis.
There is also good video across loading the Big foot data into the redis : https://www.youtube.com/watch?v=TqTg6RijfaU
Do we need to fine tune redis here.
Increase the memory for redis if it's getting OOMkilled.
We have a node.js web server that makes some outgoing http requests to an external API. It's running in docker using dokku.
After some time of load (30req/s) these outgoing requests aren't getting responses anymore.
Here's a graph I made while testing with constant req/s:
incoming and outgoing is the amount of concurrent requests (not the number of initialized requests). (It's hard to see in the graph, but it's fairly constant at ~10 requests for each.)
response time is for external requests only.
You can clearly see that they start failing all of a sudden (hitting our 1000ms timeout).
The more req/s we send, the faster we run into this problem, so we must have some sort of limit we're getting closer to with each request.
I used netstat -ant | tail -n +3 | wc -l on the host to get the number of open connections, but it was only ~450 (most of them TIME_WAIT). That shouldn't hit the socket limit. We aren't hitting any RAM or CPU limits, either.
I also tried running the same app on the same machine outside docker and it only happens in docker.
It could be due to the Docker userland proxy. If you are running a recent version of Docker, try running the daemon with the --userland-proxy=false option. This will make Docker handle port forwarding with just iptables and there is less overhead.
I am trying to see if this is possible. I will first you you a background of how the application currently runs.
The application is deployed to 4 separate nodes (using the 'all' config). 2 nodes on ServerA and 2 nodes on ServerB named node1, node2, node3, node4.
The application is behind a web server running apache and mod_jk to redirect traffic.
Assume that version 1.0.0 is currently deployed.
I will be trying to deploy 1.0.1 which will only have a minor change.
The goal will be to take down node4, deploy version 1.0.1 to node4 (while node1-node3 are still up and running).
They will be sharing the same database which in theory should be fine as long as our code doesn't require us to update anything in our database.
The next step would be to direct traffic using apache + mod_jk to only load balance node1-node3. node4 will be accessed directly.
node4 will be tested running version 1.0.1.
Apache + mod_jk will be changed to serve node4.
Version 1.0.1 will be deployed to node1-node3.
All nodes should now be running version 1.0.1.
I know this is extremely high level and I am already facing problems (not to mention application specific problems).
I just want to know what are other ways of approaching this or JBoss specific problems that I can run into.
Should I be putting the hotdeploy node in a different cluster and have the rest join later?
Any suggestions would help. Thanks.
You can take advantage of your Apache with mod_jk in front, imagine you have in your configuration something like:
JkMount /myapp/* workerApp
JkWorkersFile /etc/httpd/conf/workerApp.properties
Well instead of having a file named workerApp.properties use these 3 files:
workerApp-deploy1.properties
Will contain configuration to connect only to node 4
workerApp-deploy2.properties
Will contain configuration to connect only to nodes 1,2 and 3
workerApp-normal.properties
This will be your actual workers file
Now wokerApp.properties instead of being a file is a link, so on normal cirscunstances:
ln -s workerApp-normal.properties workerApp.properties
When you deploy a new version
rm -f workerApp.properties
ln -s workerApp-deploy2.properties workerApp.properties
reload apache
Now you can deploy on node4 the new version and all request will route through node1,2 and 3. When deploy on node 4 is ready:
rm -f workerApp.properties
ln -s workerApp-deploy1.properties workerApp.properties
reload apache
In this situation all clients will be router to node 4 and you can upgrade versions on other nodes. When you're done:
rm -f workerApp.properties
ln -s workerApp-normal.properties workerApp.properties
reload apache
And you get all request balanced between servers.
This have another advantage, for example you can define a VirtualHost like preflighttest.yourcompany.com using a different set of workers, so you can test your new version on node 4 before effectivily rolling it in production.
Hope it helps.
I am running a series of benchmarks with Cassandra. Among others, I tried the following configuration: 1 client node, 3 server nodes (same ring). All experiments are run after cleaning up the servers:
pkill -9 java; sleep 2; rm -r /var/lib/cassandra/*; ./apache-cassandra-1.2.2/bin/cassandra -f
then I run cassandra-stress from the client node (3 replica, consistency ANY/ALL):
[stop/clean/start servers]
./tools/bin/cassandra-stress -o INSERT -d server1,server2,server3 -l 3 -e ANY
[224 seconds]
[stop/clean/start servers]
./tools/bin/cassandra-stress -o INSERT -d server1,server2,server3 -l 3 -e ALL
[368 seconds]
One would deduce that decreasing the consistency level increases performance. However, there is no reason why this should happen. The bottleneck is the CPU on the servers and they all have to eventually do a local write. In fact, a careful read of the server logs reveals that hinted hand-off has taken place. Repeating the experiment, I sometimes get UnavailableException on the client and "MUTATION messages dropped" on the server.
Is this issue documented? Should CL != ALL be considered harmful on writes?
I'm not quite sure what your point is. Things appear to be working as designed.
Yes, if you're writing at CL.ONE it will complete the write faster that at CL.ALL - because it only has to get an ACK from one node - not all of them.
However, you're not measuring the time that will be taken to repair the data. You will get time spent queueing up and processing the hinted handoffs - however, nodes only hold this up for an hour.
Eventually, you'll have to run a nodetool repair to correct the consistency and delete the tombstones.