How does chronyd work when it changes from a state where the time is synchronized to the server whose name is resolved from the pool to a state where communication with the server is lost?
CentOS 7
# chronyd -v
chronyd (chrony) version 1.29.1
# grep server /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
server 0.amazon.pool.ntp.org iburst
server 1.amazon.pool.ntp.org iburst
server 2.amazon.pool.ntp.org iburst
server 3.amazon.pool.ntp.org iburst
# Serve time even if not synchronized to any NTP server.
Time was not synchronized
# chronyc sources
210 Number of sources = 4
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^? 45.76.218.213.vultr.com 2 10 0 518d -6679us[-6905us] +/- 40ms
^? v150-95-148-140.a08d.g.ty 2 10 0 745d -105.4s[ +315us] +/- 47ms
^? 45.76.98.188.vultr.com 3 10 0 564d +1031ms[ -242us] +/- 209ms
^? 153-128-30-125.compute.jp 2 10 0 748d -108.1s[-2427us] +/- 37ms
Related
I've noticed this about 4 days ago and dont know what to do right now. The problem is as follows:
I have a 6 node 3 monitor ceph cluster with 84 osds, 72x7200rpm spin disks and 12xnvme ssds for journaling. Every value for scrub configurations are the default values. Every pg in the cluster is active+clean, every cluster stat is green. Yet PGs not deep scrubbed in time keeps increasing and it is at 96 right now. Output from ceph -s:
cluster:
id: xxxxxxxxxxxxxxxxx
health: HEALTH_WARN
1 large omap objects
96 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum mon1,mon2,mon3 (age 6h)
mgr: mon2(active, since 2w), standbys: mon1
mds: cephfs:1 {0=mon2=up:active} 2 up:standby
osd: 84 osds: 84 up (since 4d), 84 in (since 3M)
rgw: 3 daemons active (mon1, mon2, mon3)
data:
pools: 12 pools, 2006 pgs
objects: 151.89M objects, 218 TiB
usage: 479 TiB used, 340 TiB / 818 TiB avail
pgs: 2006 active+clean
io:
client: 1.3 MiB/s rd, 14 MiB/s wr, 93 op/s rd, 259 op/s wr
How do i solve this problem? Also ceph health detail output shows that this non deep-scrubbed pg alerts started in january 25th but i didn't notice this before. The time I noticed this was when an osd went down for 30 seconds and got up. Might it be related to this issue? will it just resolve itself? should i tamper with the scrub configurations? For example how much performance loss i might face on client side if i increase osd_max_scrubs to 2 from 1?
Usually the cluster deep-scrubs itself during low I/O intervals on the cluster. The default is every PG has to be deep-scrubbed once a week. If OSDs go down they can't be deep-scrubbed, of course, this could cause some delay.
You could run something like this to see which PGs are behind and if they're all on the same OSD(s):
ceph pg dump pgs | awk '{print $1" "$23}' | column -t
Sort the output if necessary, and you can issue a manual deep-scrub on one of the affected PGs to see if the number decreases and if the deep-scrub itself works.
ceph pg deep-scrub <PG_ID>
Also please add ceph osd pool ls detail to see if any flags are set.
You can set the deep scrub period to 2 week, to stretch the deep scrub window.
Insted of
osd_deep_scrub_interval = 604800
use:
osd_deep_scrub_interval = 1209600
Mr. Eblock has a good idea to force manually some of the pgs for deep scrub , to spread the actions evently within 2 week.
You have 2 options:
Increase the interval between deep scrubs.
Control deep scrubbing manually with a standalone script.
I've written a simple PHP script which takes care of deep scrubbing for me: https://gist.github.com/ethaniel/5db696d9c78516308b235b0cb904e4ad
It lists all the PGs, picks 1 PG which have a last deep scrub done more than 2 weeks ago (the script takes the oldest one), checks if the OSDs that the PG sits on are not being used for another scrub (are in active+clean state), and only then starts a deep scrub on that PG. Otherwise it goes looking for another PG.
I have osd_max_scrubs set to 1 (otherwise OSD daemons start crashing due to a bug in Ceph), so this script works nicely with the regular scheduler - whichever starts the scrubbing on a PG-OSD first, wins.
i have this situation and cand access to ceph dashboard.i haad 5 mon but 2 of them went down and one of them is the bootstrap mon node so that have mgr and I got this from that node.
2020-10-14T18:59:46.904+0330 7f9d2e8e9700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_WARN
no active mgr
services:
mon: 3 daemons, quorum srv4,srv5,srv6 (age 2d)
mgr: no daemons active (since 2d)
mds: heyatfs:1 {0=heyfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 47h), 54 in (since 3w)
task status:
scrub status:
mds.heyfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 223.95k objects, 386 GiB
usage: 1.2 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 105 KiB/s rd, 328 KiB/s wr, 0 op/s rd, 0 op/s wr
I have to say the whole story, I used cephadm to create my cluster at first and I'm so new to ceph i have 15 servers and 14 of them have OSD container and 5 of them had mon and my bootstrap mon that is srv2 have mgr.
2 of these servers have public IP and I used one of them as a client (I know this structure have a lot of question in it but my company forces me to do it and also I'm new to ceph so it's how it's now). 2 weeks ago I lost 2 OSD and I said to datacenter who gives me these servers to change that 2 HDD they restart those servers and unfortunately, those servers were my Mon server. after they restarted those servers on of them came back srv5 but I could see srv3 is out of quorum
so i begon to solve this problem so I used this command in ceph shell --fsid ...
ceph orch apply mon srv3
ceph mon remove srv3
after some while I see in my dashboard srv2 my boostrap mon and mgr is not working and when I used ceph -s ssrv2 isn't there and I can see srv2 mon in removed directory
root#srv2:/var/lib/ceph/e97c1944-e132-11ea-9bdd-e83935b1c392# ls
crash crash.srv2 home mgr.srv2.xpntaf osd.0 osd.1 osd.2 osd.3 removed
but mgr.srv2.xpntaf is running and unfortunately, I lost my access to ceph dashboard now
i tried to add srv2 and 3 to monmap with
576 ceph orch daemon add mon srv2:172.32.X.3
577 history | grep dump
578 ceph mon dump
579 ceph -s
580 ceph mon dump
581 ceph mon add srv3 172.32.X.4:6789
and now
root#srv2:/# ceph -s
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_WARN
no active mgr
2/5 mons down, quorum srv4,srv5,srv6
services:
mon: 5 daemons, quorum srv4,srv5,srv6 (age 16h), out of quorum: srv2, srv3
mgr: no daemons active (since 2d)
mds: heyatfs:1 {0=heyatfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 2d), 54 in (since 3w)
task status:
scrub status:
mds.heyatfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 223.95k objects, 386 GiB
usage: 1.2 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 105 KiB/s rd, 328 KiB/s wr, 0 op/s rd, 0 op/s wr
and I must say ceph orch host ls doesn't work and it hangs when I run it and I think it's because of that err no active mgr and also when I see that removed directory mon.srv2 is there and you can see unit.run file so I used that command to run the container again but it says mon.srv2 isn't on mon map and doesn't have specific IP and by the way I must say after ceph orch apply mon srv3 i could see a new container with a new fsid in srv3 server
I now my whole problem is because I ran this command ceph orch apply mon srv3
because when you see the installation document :
To deploy monitors on a specific set of hosts:
# ceph orch apply mon *<host1,host2,host3,...>*
Be sure to include the first (bootstrap) host in this list.
and I didn't see that line !!!
now I manage to have another mgr running but I got this
root#srv2:/var/lib/ceph/mgr# ceph -s
2020-10-15T13:11:59.080+0000 7f957e9cd700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
cluster:
id: e97c1944-e132-11ea-9bdd-e83935b1c392
health: HEALTH_ERR
1 stray daemons(s) not managed by cephadm
2 mgr modules have failed
2/5 mons down, quorum srv4,srv5,srv6
services:
mon: 5 daemons, quorum srv4,srv5,srv6 (age 20h), out of quorum: srv2, srv3
mgr: srv4(active, since 8m)
mds: heyatfs:1 {0=heyatfs.srv10.lxizhc=up:active} 1 up:standby
osd: 54 osds: 54 up (since 2d), 54 in (since 3w)
task status:
scrub status:
mds.heyatfs.srv10.lxizhc: idle
data:
pools: 3 pools, 65 pgs
objects: 301.77k objects, 537 GiB
usage: 1.6 TiB used, 97 TiB / 98 TiB avail
pgs: 65 active+clean
io:
client: 180 KiB/s rd, 597 B/s wr, 0 op/s rd, 0 op/s wr
and when I run the ceph orch host ls i see this
root#srv2:/var/lib/ceph/mgr# ceph orch host ls
HOST ADDR LABELS STATUS
srv10 172.32.x.11
srv11 172.32.x.12
srv12 172.32.x.13
srv13 172.32.x.14
srv14 172.32.x.15
srv15 172.32.x.16
srv2 srv2
srv3 172.32.x.4
srv4 172.32.x.5
srv5 172.32.x.6
srv6 172.32.x.7
srv7 172.32.x.8
srv8 172.32.x.9
srv9 172.32.x.10
I have a golang program running on centos that usually has around 5k tcp clients connected. Every once in a while this number goes to around 15k for about an hour and still everything is fine.
The program has a slow shutdown mode where it stops accepting new clients and slowly kills all currently connected clients over the course of 20 mins. During these slow shutdown periods if the machine has 15k clients, sometimes I get:
Wed Oct 31 21:28:23 2018] net_ratelimit: 482 callbacks suppressed
[Wed Oct 31 21:28:23 2018] TCP: too many orphaned sockets
[Wed Oct 31 21:28:23 2018] TCP: too many orphaned sockets
[Wed Oct 31 21:28:23 2018] TCP: too many orphaned sockets
I have tried adding:
echo "net.ipv4.tcp_max_syn_backlog=5000" >> /etc/sysctl.conf
echo "net.ipv4.tcp_fin_timeout=10" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_recycle=1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_reuse=1" >> /etc/sysctl.conf
sysctl -f /etc/sysctl.conf
And these values are set, I see them with their correct new values. A typical sockstat is:
cat /proc/net/sockstat
sockets: used 31682
TCP: inuse 17286 orphan 5 tw 3874 alloc 31453 mem 15731
UDP: inuse 8 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
And ideas how to stop the too many orphaned socket error and crash? Should I increase the 20 min slow shutdown period to 40 mins? Increase tcp_mem? Thanks!
On my local I run a mysql container and then ping it from another container on the same network:
$ docker run -d tutum/mysql
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID
PING 67e35427d638 (198.105.244.24): 56 data bytes
64 bytes from 198.105.244.24: icmp_seq=0 ttl=37 time=0.243 ms
...
That is good. Then, using Kubernetes(minikube) locally, I deploy tutum/mysql using the following YAML:
...
- name: mysql
image: tutum/mysql
...
There is nothing else for the mysql container. Then I deploy it, ssh into the minikube pod, spin up a random container and try pinging the mysql container inside the pod this time:
$ kubectl create -f k8s-deployment.yml
$ minikube ssh
$ docker ps
$ docker run -it plumsempy/plum bash
# ping MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
PING mysql (198.105.244.24): 56 data bytes
^C--- mysql ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
# traceroute MYSQL_CONTAINER_ID_INSIDE_MINIKUBE
traceroute to aa7f7ed7af01 (198.105.244.24), 30 hops max, 60 byte packets
1 172.17.0.1 (172.17.0.1) 0.031 ms 0.009 ms 0.007 ms
2 10.0.2.2 (10.0.2.2) 0.156 ms 0.086 ms 0.050 ms
3 * * *
4 * * *
5 dtr02gldlca-tge-0-2-0-1.gldl.ca.charter.com (96.34.102.201) 16.153 ms 16.107 ms 16.077 ms
6 crr01lnbhca-bue-200.lnbh.ca.charter.com (96.34.98.188) 18.753 ms 18.011 ms 30.642 ms
7 crr01mtpkca-bue-201.mtpk.ca.charter.com (96.34.96.63) 30.779 ms 30.523 ms 30.428 ms
8 bbr01mtpkca-bue-2.mtpk.ca.charter.com (96.34.2.24) 24.089 ms 23.900 ms 23.814 ms
9 bbr01ashbva-tge-0-1-0-1.ashb.va.charter.com (96.34.3.139) 26.061 ms 25.949 ms 36.002 ms
10 10ge9-10.core1.lax1.he.net (65.19.189.177) 34.027 ms 34.436 ms 33.857 ms
11 100ge12-1.core1.ash1.he.net (184.105.80.201) 107.873 ms 107.750 ms 104.078 ms
12 100ge3-1.core1.nyc4.he.net (184.105.223.166) 100.554 ms 100.478 ms 100.393 ms
13 xerocole-inc.10gigabitethernet12-4.core1.nyc4.he.net (216.66.41.242) 109.184 ms 111.122 ms 111.018 ms
14 * * *
15 * * *
...(til it ends)
the plumsempy/plum can be any container since they are both on the same network and same pod, the pinging should go through. The question is Why can I not reach mysql on minikube and how could I fix that?
From k8s multi-container pod docs:
Pods share fate, and share some resources, such as storage volumes and IP addresses.
Hence the mysql container is reachable from the plum container at the IP address 127.0.0.1.
Also, since mysql runs on port 3306 by default, you probably want telnet 127.0.0.1 3306 to check if it's reachable (ping uses ICMP which doesn't have the concept of ports).
I guess the container ID just don't work with Kubernetes. You can also see, that the container ID resolved to the public IP 198.105.244.24, which looks wrong.
You have multiple ways to contact this pod:
get the pod IP via kubectl describe -f k8s-deployment.yml
create a service for that pod and do one of these (assuming the service name is mysql):
use environment variables like ping ${MYSQL_SERVICE_HOST}
use DNS like ping mysql.default.svc.cluster.local
Running this command in the mongodb installation file from mongodb.org
./mongo ds045907.mongolab.com:45907/database -u user -p password
I changed Database, user, and password for anonymity.
results in this
Error: couldn't connect to server ds045907.mongolab.com:45907 src/mongo/shell/mongo.js:93
exception: connect failed
Maybe i'm being blocked by a server firewall? I have no problem using git or brew or pip...
Here are a few things you can try, but you can always feel free to contact us at support#mongolab.com. I'm sure we can get to the bottom of this.
Anonymous mongo shell connection
Mongo will let you connect without authenticating. You can do very little with an unauthenticated connection, but you can use it as a test to separate a connectivity problem from a credentials problem.
% mongo ds045907.mongolab.com:45907
MongoDB shell version: 2.0.7
connecting to: ds045907.mongolab.com:45907/test
> db.version()
2.2.2
> db.runCommand({ping:1})
{ "ok" : 1 }
> exit
bye
If you can connect without authenticating and run the commands as shown above, but trying to connect with authentication fails, then you have a problem with the credentials. If, however, connecting doesn't work even without supplying credentials then you have a connectivity problem.
ping
That server does allow ICMP traffic, so make sure it's reachable from wherever you are.
% ping ds045907.mongolab.com
PING ec2-107-20-85-188.compute-1.amazonaws.com (107.20.85.188): 56 data bytes
64 bytes from 107.20.85.188: icmp_seq=0 ttl=41 time=99.744 ms
64 bytes from 107.20.85.188: icmp_seq=1 ttl=41 time=99.475 ms
64 bytes from 107.20.85.188: icmp_seq=2 ttl=41 time=99.930 ms
^C
--- ec2-107-20-85-188.compute-1.amazonaws.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 99.475/99.716/99.930/0.187 ms
traceroute
If ping fails, use traceroute (or tracert on Windows) to try to figure out where the problem is. Once the trace reaches AWS, however, it will trail off. That's normal. AWS prevents traces from seeing too far into their networks. Make sure that the last IP on your list is owned by Amazon using some kind of IP reverse lookup tool (many on the Web).
% traceroute ds045907.mongolab.com
traceroute to ec2-107-20-85-188.compute-1.amazonaws.com (107.20.85.188), 64 hops max, 52 byte packets
1 192.168.1.1 (192.168.1.1) 1.092 ms 0.865 ms 1.047 ms
2 192.168.27.1 (192.168.27.1) 1.414 ms 1.330 ms 1.224 ms
... snipped to protect the innocent ...
14 72.21.220.83 (72.21.220.83) 87.777 ms
72.21.220.75 (72.21.220.75) 87.406 ms
205.251.229.55 (205.251.229.55) 99.363 ms
15 72.21.222.145 (72.21.222.145) 87.703 ms
178.236.3.24 (178.236.3.24) 98.662 ms
72.21.220.75 (72.21.220.75) 87.708 ms
16 216.182.224.55 (216.182.224.55) 87.312 ms 86.791 ms 89.005 ms
17 * 216.182.224.55 (216.182.224.55) 91.373 ms *
18 216.182.224.55 (216.182.224.55) 121.754 ms * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
It's a connection problem at your side. I tried it but got a login failure message:
MongoDB shell version: 1.6.5
connecting to: ds045907.mongolab.com:45907/database
Mon Dec 24 01:12:31 uncaught exception: login failed
exception: login failed