I got error with the latest heapster version: v1.5.1. I've described in detail in this github issue link: https://github.com/kubernetes/heapster/issues/1969
The error message:
Container Timestamp Message
heapster Mar 2, 2018, 11:03:20 AM /go/src/k8s.io/heapster/metrics/heapster.go:89 +0x458
heapster Mar 2, 2018, 11:03:20 AM main.main()
heapster Mar 2, 2018, 11:03:20 AM /go/src/k8s.io/heapster/metrics/heapster.go:194 +0x8d
heapster Mar 2, 2018, 11:03:20 AM main.createAndInitSinksOrDie(0xc42028f3b0, 0x1, 0x1, 0x0, 0x0, 0x4a817c800, 0x0, 0x0, 0xc420154740, 0x0, ...)
heapster Mar 2, 2018, 11:03:20 AM /go/src/k8s.io/heapster/metrics/sinks/factory.go:90 +0x563
heapster Mar 2, 2018, 11:03:20 AM k8s.io/heapster/metrics/sinks.(*SinkFactory).BuildAll(0xc4205e1cd8, 0xc42028f3b0, 0x1, 0x1, 0x0, 0x0, 0x7fb87afb8400, 0x0, 0x0, 0x0, ...)
heapster Mar 2, 2018, 11:03:20 AM goroutine 1 [running]:
heapster Mar 2, 2018, 11:03:20 AM
heapster Mar 2, 2018, 11:03:20 AM [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x15f77a3]
heapster Mar 2, 2018, 11:03:20 AM panic: runtime error: invalid memory address or nil pointer dereference
heapster Mar 2, 2018, 11:03:20 AM I0302 04:03:20.258025 1 configs.go:62] Using kubelet port 10255
heapster Mar 2, 2018, 11:03:20 AM I0302 04:03:20.258013 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
heapster Mar 2, 2018, 11:03:20 AM I0302 04:03:20.257857 1 heapster.go:79] Heapster version v1.5.1
heapster Mar 2, 2018, 11:03:20 AM I0302 04:03:20.257820 1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default --sink="statsd:udp://dd-agent-service.default:8125"
Anybody knows how to solve it? Perhaps someone who already successfully integrated the heapster to datadog statsd agent in Kubernetes?
Thanks before
Related
I am using Kubeadm 1.19.3 version & Flannel as cni.
I do not see any resource overutilization on my underlying node.
Below is the Kubelet log around the same timestamp:
Jan 07 18:35:53 vmcipind83wsfd2 kubelet[7593]: I0107 18:35:53.667179
7593 log.go:181] http: superfluous response.WriteHeader call from
k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader
(httplog.go:217) Jan 07 18:36:56 vmcipind83wsfd2 kubelet[7593]: I0107
18:36:56.756426 7593 topology_manager.go:219] [topologymanager]
RemoveContainer - Container ID:
32346f17b258c75f77dba18796414a51b58c442c16d66a18a6f1ca32002c0bb8 Jan
07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.273923 7593
reconciler.go:196] operationExecutor.UnmountVolume started for volume
"default-token-m9l42" (UniqueName:
"kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42")
pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID:
"3516986a-f43b-498e-82e9-4ca11a3c3cd4") Jan 07 18:37:03
vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.273966 7593
reconciler.go:196] operationExecutor.UnmountVolume started for volume
"wta" (UniqueName:
"kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv")
pod "3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID:
"3516986a-f43b-498e-82e9-4ca11a3c3cd4") Jan 07 18:37:03
vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.335585 7593
operation_generator.go:788] UnmountVolume.TearDown succeeded for
volume
"kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42"
(OuterVolumeSpecName: "default-token-m9l42") pod
"3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID:
"3516986a-f43b-498e-82e9-4ca11a3c3cd4"). InnerVolumeSpecName
"default-token-m9l42". PluginName "kubernetes.io/secret",
VolumeGidValue "" Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107
18:37:03.374229 7593 reconciler.go:319] Volume detached for volume
"default-token-m9l42" (UniqueName:
"kubernetes.io/secret/3516986a-f43b-498e-82e9-4ca11a3c3cd4-default-token-m9l42")
on node "vmcipind83wsfd2" DevicePath "" Jan 07 18:37:03
vmcipind83wsfd2 kubelet[7593]: I0107 18:37:03.803952 7593
operation_generator.go:788] UnmountVolume.TearDown succeeded for
volume
"kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv"
(OuterVolumeSpecName: "wta") pod
"3516986a-f43b-498e-82e9-4ca11a3c3cd4" (UID:
"3516986a-f43b-498e-82e9-4ca11a3c3cd4"). InnerVolumeSpecName
"wta-sun-trunk-wta-pv". PluginName "kubernetes.io/nfs", VolumeGidValue
"" Jan 07 18:37:03 vmcipind83wsfd2 kubelet[7593]: I0107
18:37:03.893440 7593 reconciler.go:319] Volume detached for volume
"wta-sun-trunk-wta-pv" (UniqueName:
"kubernetes.io/nfs/3516986a-f43b-498e-82e9-4ca11a3c3cd4-wta-sun-trunk-wta-pv")
on node "vmcipind83wsfd2" DevicePath ""
i followed this tutorial to install freeradius and dalo radius for the raspberry pi:
http://www.binaryheartbeat.net/2013/12/raspberry-pi-based-freeradius-server.html
i tested the file authentication and it worked fine but after installing daloradius and switching to MySQL authnetications fail for unknown reasons
here is freeradius output that occurs when trying to authenticate a user:
rad_recv: Access-Request packet from host 192.168.1.1 port 32779, id=216, length=172
User-Name = "ccc"
State = 0xf9775519ff7f4c9188c14494359a170f
EAP-Message = 0x0208005b190017030100500d2898ca35aa9fa9e4febd8816c9e6deda71960fe5692b7c3d0499f2b5bba6b531483e373e14f8aff517aa081e214edc98e2c8bb22d16a961ecff4f498d20d152535b4d11ace1484b985bd2501ade77b
Service-Type = Framed-User
Framed-MTU = 1420
NAS-IP-Address = 192.168.1.1
Message-Authenticator = 0x49fc781b8a152fbec467b2c1f275a1a1
Tue Dec 29 18:38:47 2015 : Info: # Executing section authorize from file /etc/freeradius/sites-enabled/default
Tue Dec 29 18:38:47 2015 : Info: +group authorize {
Tue Dec 29 18:38:47 2015 : Info: ++[preprocess] = ok
Tue Dec 29 18:38:47 2015 : Info: ++[chap] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[mschap] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[digest] = noop
Tue Dec 29 18:38:47 2015 : Info: [suffix] No '#' in User-Name = "ccc", looking up realm NULL
Tue Dec 29 18:38:47 2015 : Info: [suffix] No such realm "NULL"
Tue Dec 29 18:38:47 2015 : Info: ++[suffix] = noop
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP packet type response id 8 length 91
Tue Dec 29 18:38:47 2015 : Info: [eap] Continuing tunnel setup.
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = ok
Tue Dec 29 18:38:47 2015 : Info: +} # group authorize = ok
Tue Dec 29 18:38:47 2015 : Info: Found Auth-Type = EAP
Tue Dec 29 18:38:47 2015 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Tue Dec 29 18:38:47 2015 : Info: +group authenticate {
Tue Dec 29 18:38:47 2015 : Info: [eap] Request found, released from the list
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP/peap
Tue Dec 29 18:38:47 2015 : Info: [eap] processing type peap
Tue Dec 29 18:38:47 2015 : Info: [peap] processing EAP-TLS
Tue Dec 29 18:38:47 2015 : Info: [peap] eaptls_verify returned 7
Tue Dec 29 18:38:47 2015 : Info: [peap] Done initial handshake
Tue Dec 29 18:38:47 2015 : Info: [peap] eaptls_process returned 7
Tue Dec 29 18:38:47 2015 : Info: [peap] EAPTLS_OK
Tue Dec 29 18:38:47 2015 : Info: [peap] Session established. Decoding tunneled attributes.
Tue Dec 29 18:38:47 2015 : Info: [peap] Peap state phase2
Tue Dec 29 18:38:47 2015 : Info: [peap] EAP type mschapv2
Tue Dec 29 18:38:47 2015 : Info: [peap] Got tunneled request
EAP-Message = 0x0208003e1a0208003931461c2f1334a4b7bab38912e9d82dd97b000000000000000070fb7810a938a00d884f17dc01b62eaa7dde9fbb7ab2cf4200636363
server {
Tue Dec 29 18:38:47 2015 : Info: [peap] Setting User-Name to ccc
Sending tunneled request
EAP-Message = 0x0208003e1a0208003931461c2f1334a4b7bab38912e9d82dd97b000000000000000070fb7810a938a00d884f17dc01b62eaa7dde9fbb7ab2cf4200636363
FreeRADIUS-Proxied-To = 127.0.0.1
User-Name = "ccc"
State = 0x4bb6eef44bbef48a7072f4e023895561
server inner-tunnel {
Tue Dec 29 18:38:47 2015 : Info: # Executing section authorize from file /etc/freeradius/sites-enabled/inner-tunnel
Tue Dec 29 18:38:47 2015 : Info: +group authorize {
Tue Dec 29 18:38:47 2015 : Info: ++[chap] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[mschap] = noop
Tue Dec 29 18:38:47 2015 : Info: [suffix] No '#' in User-Name = "ccc", looking up realm NULL
Tue Dec 29 18:38:47 2015 : Info: [suffix] No such realm "NULL"
Tue Dec 29 18:38:47 2015 : Info: ++[suffix] = noop
Tue Dec 29 18:38:47 2015 : Info: ++update control {
Tue Dec 29 18:38:47 2015 : Info: ++} # update control = noop
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP packet type response id 8 length 62
Tue Dec 29 18:38:47 2015 : Info: [eap] No EAP Start, assuming it's an on-going EAP conversation
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = updated
Tue Dec 29 18:38:47 2015 : Info: ++[files] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[expiration] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[logintime] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[pap] = noop
Tue Dec 29 18:38:47 2015 : Info: +} # group authorize = updated
Tue Dec 29 18:38:47 2015 : Info: Found Auth-Type = EAP
Tue Dec 29 18:38:47 2015 : Info: # Executing group from file /etc/freeradius/sites-enabled/inner-tunnel
Tue Dec 29 18:38:47 2015 : Info: +group authenticate {
Tue Dec 29 18:38:47 2015 : Info: [eap] Request found, released from the list
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP/mschapv2
Tue Dec 29 18:38:47 2015 : Info: [eap] processing type mschapv2
Tue Dec 29 18:38:47 2015 : Info: [mschapv2] # Executing group from file /etc/freeradius/sites-enabled/inner-tunnel
Tue Dec 29 18:38:47 2015 : Info: [mschapv2] +group MS-CHAP {
Tue Dec 29 18:38:47 2015 : Info: [mschap] No Cleartext-Password configured. Cannot create LM-Password.
Tue Dec 29 18:38:47 2015 : Info: [mschap] No Cleartext-Password configured. Cannot create NT-Password.
Tue Dec 29 18:38:47 2015 : Info: [mschap] Creating challenge hash with username: ccc
Tue Dec 29 18:38:47 2015 : Info: [mschap] Client is using MS-CHAPv2 for ccc, we need NT-Password
Tue Dec 29 18:38:47 2015 : Info: [mschap] FAILED: No NT/LM-Password. Cannot perform authentication.
Tue Dec 29 18:38:47 2015 : Info: [mschap] FAILED: MS-CHAP2-Response is incorrect
Tue Dec 29 18:38:47 2015 : Info: ++[mschap] = reject
Tue Dec 29 18:38:47 2015 : Info: +} # group MS-CHAP = reject
Tue Dec 29 18:38:47 2015 : Info: [eap] Freeing handler
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = reject
Tue Dec 29 18:38:47 2015 : Info: +} # group authenticate = reject
Tue Dec 29 18:38:47 2015 : Info: Failed to authenticate the user.
Tue Dec 29 18:38:47 2015 : Info: Using Post-Auth-Type REJECT
Tue Dec 29 18:38:47 2015 : Info: # Executing group from file /etc/freeradius/sites-enabled/inner-tunnel
Tue Dec 29 18:38:47 2015 : Info: +group REJECT {
Tue Dec 29 18:38:47 2015 : Info: [attr_filter.access_reject] expand: %{User-Name} -> ccc
Tue Dec 29 18:38:47 2015 : Debug: attr_filter: Matched entry DEFAULT at line 11
Tue Dec 29 18:38:47 2015 : Info: ++[attr_filter.access_reject] = updated
Tue Dec 29 18:38:47 2015 : Info: +} # group REJECT = updated
} # server inner-tunnel
Tue Dec 29 18:38:47 2015 : Info: [peap] Got tunneled reply code 3
MS-CHAP-Error = "\010E=691 R=1"
EAP-Message = 0x04080004
Message-Authenticator = 0x00000000000000000000000000000000
Tue Dec 29 18:38:47 2015 : Info: [peap] Got tunneled reply RADIUS code 3
MS-CHAP-Error = "\010E=691 R=1"
EAP-Message = 0x04080004
Message-Authenticator = 0x00000000000000000000000000000000
Tue Dec 29 18:38:47 2015 : Info: [peap] Tunneled authentication was rejected.
Tue Dec 29 18:38:47 2015 : Info: [peap] FAILURE
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = handled
Tue Dec 29 18:38:47 2015 : Info: +} # group authenticate = handled
Sending Access-Challenge of id 216 to 192.168.1.1 port 32779
EAP-Message = 0x0109002b190017030100205991bfd8f9e7f70794477d653c848e8b443626b3b935a5b3f049ac7af1534d3e
Message-Authenticator = 0x00000000000000000000000000000000
State = 0xf9775519fe7e4c9188c14494359a170f
Tue Dec 29 18:38:47 2015 : Info: Finished request 7.
Tue Dec 29 18:38:47 2015 : Debug: Going to the next request
Tue Dec 29 18:38:47 2015 : Debug: Waking up in 0.4 seconds.
rad_recv: Access-Request packet from host 192.168.1.1 port 32779, id=217, length=124
User-Name = "ccc"
State = 0xf9775519fe7e4c9188c14494359a170f
EAP-Message = 0x0209002b190017030100202a7f1a72de2970b689e44c005661d1e1e444854af7499ebeb23eabc7bfad7b64
Service-Type = Framed-User
Framed-MTU = 1420
NAS-IP-Address = 192.168.1.1
Message-Authenticator = 0xc9b0d8e268df2d8e4b484725c3efa189
Tue Dec 29 18:38:47 2015 : Info: # Executing section authorize from file /etc/freeradius/sites-enabled/default
Tue Dec 29 18:38:47 2015 : Info: +group authorize {
Tue Dec 29 18:38:47 2015 : Info: ++[preprocess] = ok
Tue Dec 29 18:38:47 2015 : Info: ++[chap] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[mschap] = noop
Tue Dec 29 18:38:47 2015 : Info: ++[digest] = noop
Tue Dec 29 18:38:47 2015 : Info: [suffix] No '#' in User-Name = "ccc", looking up realm NULL
Tue Dec 29 18:38:47 2015 : Info: [suffix] No such realm "NULL"
Tue Dec 29 18:38:47 2015 : Info: ++[suffix] = noop
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP packet type response id 9 length 43
Tue Dec 29 18:38:47 2015 : Info: [eap] Continuing tunnel setup.
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = ok
Tue Dec 29 18:38:47 2015 : Info: +} # group authorize = ok
Tue Dec 29 18:38:47 2015 : Info: Found Auth-Type = EAP
Tue Dec 29 18:38:47 2015 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Tue Dec 29 18:38:47 2015 : Info: +group authenticate {
Tue Dec 29 18:38:47 2015 : Info: [eap] Request found, released from the list
Tue Dec 29 18:38:47 2015 : Info: [eap] EAP/peap
Tue Dec 29 18:38:47 2015 : Info: [eap] processing type peap
Tue Dec 29 18:38:47 2015 : Info: [peap] processing EAP-TLS
Tue Dec 29 18:38:47 2015 : Info: [peap] eaptls_verify returned 7
Tue Dec 29 18:38:47 2015 : Info: [peap] Done initial handshake
Tue Dec 29 18:38:47 2015 : Info: [peap] eaptls_process returned 7
Tue Dec 29 18:38:47 2015 : Info: [peap] EAPTLS_OK
Tue Dec 29 18:38:47 2015 : Info: [peap] Session established. Decoding tunneled attributes.
Tue Dec 29 18:38:47 2015 : Info: [peap] Peap state send tlv failure
Tue Dec 29 18:38:47 2015 : Info: [peap] Received EAP-TLV response.
Tue Dec 29 18:38:47 2015 : Info: [peap] The users session was previously rejected: returning reject (again.)
Tue Dec 29 18:38:47 2015 : Info: [peap] *** This means you need to read the PREVIOUS messages in the debug output
Tue Dec 29 18:38:47 2015 : Info: [peap] *** to find out the reason why the user was rejected.
Tue Dec 29 18:38:47 2015 : Info: [peap] *** Look for "reject" or "fail". Those earlier messages will tell you.
Tue Dec 29 18:38:47 2015 : Info: [peap] *** what went wrong, and how to fix the problem.
Tue Dec 29 18:38:47 2015 : Info: [eap] Handler failed in EAP/peap
Tue Dec 29 18:38:47 2015 : Info: [eap] Failed in EAP select
Tue Dec 29 18:38:47 2015 : Info: ++[eap] = invalid
Tue Dec 29 18:38:47 2015 : Info: +} # group authenticate = invalid
Tue Dec 29 18:38:47 2015 : Info: Failed to authenticate the user.
Tue Dec 29 18:38:47 2015 : Info: Using Post-Auth-Type REJECT
Tue Dec 29 18:38:47 2015 : Info: # Executing group from file /etc/freeradius/sites-enabled/default
Tue Dec 29 18:38:47 2015 : Info: +group REJECT {
Tue Dec 29 18:38:47 2015 : Info: [sql] expand: %{User-Name} -> ccc
Tue Dec 29 18:38:47 2015 : Info: [sql] sql_set_user escaped user --> 'ccc'
Tue Dec 29 18:38:47 2015 : Info: [sql] expand: %{User-Password} ->
Tue Dec 29 18:38:47 2015 : Info: [sql] ... expanding second conditional
Tue Dec 29 18:38:47 2015 : Info: [sql] expand: %{Chap-Password} ->
Tue Dec 29 18:38:47 2015 : Info: [sql] expand: INSERT INTO radpostauth (username, pass, reply, authdate) VALUES ( '%{User-Name}', '%{%{User-Password}:-%{Chap-Password}}', '%{reply:Packet-Type}', '%S') -> INSERT INTO radpostauth (username, pass, reply, authdate) VALUES ( 'ccc', '', 'Access-Reject', '2015-12-29 18:38:47')
Tue Dec 29 18:38:47 2015 : Debug: rlm_sql (sql) in sql_postauth: query is INSERT INTO radpostauth (username, pass, reply, authdate) VALUES ( 'ccc', '', 'Access-Reject', '2015-12-29 18:38:47')
Tue Dec 29 18:38:47 2015 : Debug: rlm_sql (sql): Reserving sql socket id: 29
Tue Dec 29 18:38:47 2015 : Debug: rlm_sql (sql): Released sql socket id: 29
Tue Dec 29 18:38:47 2015 : Info: ++[sql] = ok
Tue Dec 29 18:38:47 2015 : Info: [attr_filter.access_reject] expand: %{User-Name} -> ccc
Tue Dec 29 18:38:47 2015 : Debug: attr_filter: Matched entry DEFAULT at line 11
Tue Dec 29 18:38:47 2015 : Info: ++[attr_filter.access_reject] = updated
Tue Dec 29 18:38:47 2015 : Info: +} # group REJECT = updated
Tue Dec 29 18:38:47 2015 : Info: Delaying reject of request 8 for 1 seconds
Tue Dec 29 18:38:47 2015 : Debug: Going to the next request
Tue Dec 29 18:38:47 2015 : Debug: Waking up in 0.1 seconds.
Tue Dec 29 18:38:47 2015 : Info: Cleaning up request 0 ID 209 with timestamp +11
Tue Dec 29 18:38:47 2015 : Debug: Waking up in 0.3 seconds.
Tue Dec 29 18:38:47 2015 : Info: Cleaning up request 1 ID 210 with timestamp +11
Tue Dec 29 18:38:47 2015 : Debug: Waking up in 0.3 seconds.
Tue Dec 29 18:38:48 2015 : Info: Cleaning up request 2 ID 211 with timestamp +12
Tue Dec 29 18:38:48 2015 : Debug: Waking up in 0.1 seconds.
Tue Dec 29 18:38:48 2015 : Info: Sending delayed reject for request 8
Sending Access-Reject of id 217 to 192.168.1.1 port 32779
EAP-Message = 0x04090004
Message-Authenticator = 0x00000000000000000000000000000000
Found the solution,
the problem was that i didn't configure the /etc/raddb/sites-available/inner-tunnel file to use sql
fedora :
[root#host3 vagrant]# cat /etc/os-release
NAME=Fedora
VERSION="23 (Twenty Three)"
ID=fedora
VERSION_ID=23
PRETTY_NAME="Fedora 23 (Twenty Three)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:23"
HOME_URL="https://fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=23
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=23
PRIVACY_POLICY_URL=https://fedoraproject.org/wiki/Legal:PrivacyPolicy
kube-proxy version:
[root#host3 vagrant]# kube-proxy --version=true
Kubernetes v1.1.2
Run command and error msg:
[root#host3 vagrant]# kube-proxy --logtostderr=true --v=0 --master=http://host1:8080 --proxy-mode=userspace --cleanup-iptables=true
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xb1 pc=0x465e26]
goroutine 1 [running]:
k8s.io/kubernetes/cmd/kube-proxy/app.(*ProxyServer).Run(0xc2080d79d0, 0xc208046af0, 0x0, 0x5, 0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/app/server.go:309 +0x56
main.main()
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-proxy/proxy.go:53 +0x225
goroutine 7 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x12169e0)
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/golang/glog.init·1
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x2a7
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/usr/src/go/src/runtime/asm_amd64.s:2232 +0x1
goroutine 15 [chan receive]:
github.com/godbus/dbus.(*Conn).outWorker(0xc2080daa20)
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:367 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:119 +0xea1
goroutine 12 [sleep]:
k8s.io/kubernetes/pkg/util.Until(0xda66f0, 0x12a05f200, 0xc20800a7e0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/util.go:127 +0x98
created by k8s.io/kubernetes/pkg/util.InitLogs
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/logs.go:49 +0xab
goroutine 14 [IO wait]:
net.(*pollDesc).Wait(0xc2080d6920, 0x72, 0x0, 0x0)
/usr/src/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2080d6920, 0x0, 0x0)
/usr/src/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).readMsg(0xc2080d68c0, 0xc2080e28c0, 0x10, 0x10, 0xc20816d220, 0x1000, 0x1000, 0xffffffffffffffff, 0x0, 0x0, ...)
/usr/src/go/src/net/fd_unix.go:296 +0x54e
net.(*UnixConn).ReadMsgUnix(0xc20803a0f0, 0xc2080e28c0, 0x10, 0x10, 0xc20816d220, 0x1000, 0x1000, 0x0, 0xc2080e276c, 0x4, ...)
/usr/src/go/src/net/unixsock_posix.go:147 +0x167
github.com/godbus/dbus.(*oobReader).Read(0xc20816d200, 0xc2080e28c0, 0x10, 0x10, 0xc20816d200, 0x0, 0x0)
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:21 +0xc5
io.ReadAtLeast(0x7f7eeae0df58, 0xc20816d200, 0xc2080e28c0, 0x10, 0x10, 0x10, 0x0, 0x0, 0x0)
/usr/src/go/src/io/io.go:298 +0xf1
io.ReadFull(0x7f7eeae0df58, 0xc20816d200, 0xc2080e28c0, 0x10, 0x10, 0x0, 0x0, 0x0)
/usr/src/go/src/io/io.go:316 +0x6d
github.com/godbus/dbus.(*unixTransport).ReadMessage(0xc2081250d0, 0xc208112660, 0x0, 0x0)
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/transport_unix.go:85 +0x1bf
github.com/godbus/dbus.(*Conn).inWorker(0xc2080daa20)
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/conn.go:241 +0x58
created by github.com/godbus/dbus.(*Conn).Auth
/go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/godbus/dbus/auth.go:118 +0xe84
goroutine 16 [runnable]:
k8s.io/kubernetes/pkg/util/iptables.(*runner).dbusSignalHandler(0xc2080d6850, 0x7f7eeae0e028, 0xc20803a100)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:525
created by k8s.io/kubernetes/pkg/util/iptables.(*runner).connectToFirewallD
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/iptables/iptables.go:186 +0x7a7
anyone who can help me?
This looks like a bug when using the --cleanup-iptables=true flag in the 1.1.2 release, as I can reproduce a panic when running on a GCE node. I've created kubernetes#18197 on your behalf and this bug will be fixed in the upcoming 1.1.3 release.
I've been trying to get the Nginx newrelic nginx plugin installed and working: http://newrelic.com/plugins/nginx-inc/13
Although it seems to be installed and running correctly, no data appears in the newrelic console.
I know that the nginx module is configured and working fine and the agent appears to be working as expected, and newrelic has recognized that the plugin is running and created a plugin section in the console.
When I run ./newrelic_nginx_agent in order to insure that there's no problems with the plugins I get output like this:
[root#nginx1:newrelic_nginx_agent]$ ./newrelic_nginx_agent
Faraday: you may want to install system_timer for reliable timeouts
[Fri Aug 02 21:01:27 UTC 2013] Agent Nginx is at version 1.0.1
active=39 accepts=4098 handled=4098 requests=12502 reading=20 writing=3 waiting=16
[Fri Aug 02 21:01:27 UTC 2013] Metrics for nginx1[com.nginx.newrelic-agent] for last 60 seconds:
[Fri Aug 02 21:01:28 UTC 2013] Sent 4 metrics to New Relic [https://platform-api.newrelic.com]:
[Fri Aug 02 21:01:28 UTC 2013] Component/Connections/Active[Connections]: count: 1, total: 39.0, min: 39.0, max: 39.0, sum_of_squares: 1521.0
[Fri Aug 02 21:01:28 UTC 2013] Component/Connections/Reading[Connections]: count: 1, total: 20.0, min: 20.0, max: 20.0, sum_of_squares: 400.0
[Fri Aug 02 21:01:28 UTC 2013] Component/Connections/Writing[Connections]: count: 1, total: 3.0, min: 3.0, max: 3.0, sum_of_squares: 9.0
[Fri Aug 02 21:01:28 UTC 2013] Component/Connections/Waiting[Connections]: count: 1, total: 16.0, min: 16.0, max: 16.0, sum_of_squares: 256.0
[Fri Aug 02 21:01:28 UTC 2013] Gathered 4 statistics
A few seconds later I'll get additional output:
active=42 accepts=4176 handled=4176 requests=12735 reading=18 writing=1 waiting=23
[Fri Aug 02 21:03:27 UTC 2013] Metrics for nginx1[com.nginx.newrelic-agent] for last 60 seconds:
[Fri Aug 02 21:03:28 UTC 2013] Sent 6 metrics to New Relic [https://platform-api.newrelic.com]:
[Fri Aug 02 21:03:28 UTC 2013] Component/Connections/Active[Connections]: count: 1, total: 42.0, min: 42.0, max: 42.0, sum_of_squares: 1764.0
[Fri Aug 02 21:03:28 UTC 2013] Component/Connections/Reading[Connections]: count: 1, total: 18.0, min: 18.0, max: 18.0, sum_of_squares: 324.0
[Fri Aug 02 21:03:28 UTC 2013] Component/Connections/Writing[Connections]: count: 1, total: 1.0, min: 1.0, max: 1.0, sum_of_squares: 1.0
[Fri Aug 02 21:03:28 UTC 2013] Component/Connections/Waiting[Connections]: count: 1, total: 23.0, min: 23.0, max: 23.0, sum_of_squares: 529.0
[Fri Aug 02 21:03:28 UTC 2013] Component/Requests[Requests/Seconds]: count: 1, total: 1.8657934131562, min: 1.8657934131562, max: 1.8657934131562, sum_of_squares: 3.48118506057705
[Fri Aug 02 21:03:28 UTC 2013] Component/Accepts[Connections/Seconds]: count: 1, total: 0.699672564900836, min: 0.699672564900836, max: 0.699672564900836, sum_of_squares: 0.489541698074914
[Fri Aug 02 21:03:28 UTC 2013] Gathered 6 statistics
And so on...
I haven't found any forum or git repo or anyplace I can get support on this. Newrelic states that nginx would support it, but there's no support forum or even a link in regards to this plugin.
Does anyone have any ideas on how this might be debugged or fixed?
Perhaps I was just too impatient. After reloading, I now see data in the console.
What does this error mean?
Nov 8 17:57:10 unknown configd[25] <Notice>: jetsam: kernel memory event (91), free: 670, active: 2840, inactive: 2253, purgeable: 0, wired: 15143
Nov 8 17:57:11 unknown SpringBoard[92] <Warning>: Received memory warning. Level=2
Nov 8 17:57:11 unknown MobileMail[6432] <Warning>: Received memory warning. Level=2
Nov 8 17:57:11 unknown configd[25] <Notice>: jetsam: kernel termination snapshot being created
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.apple.mobileipod[0xe0a1]) Exited: Killed: 9
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:com.apple.mobilemail[0xbebe]) Exited: Killed: 9
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:paradigm.PaymentSystemiPad[0x6c80]) Bug: launchd_core_logic.c:3795 (24506):0
Nov 8 17:57:11 unknown DTMobileIS[6377] <Warning>: _memoryNotification : <NSThread: 0x1c5361d0>{name = (null), num = 1}
Nov 8 17:57:11 unknown com.apple.debugserver-50[6480] <Warning>: 1 [1950/1503]: error: ::read ( 4, 0x2ff669f0, 1024 ) => -1 err = Bad file descriptor (0x00000009)
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:paradigm.PaymentSystemiPad[0x6c80]) Bug: launchd_core_logic.c:3794 (24506):3
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:paradigm.PaymentSystemiPad[0x6c80]) Bug: launchd_core_logic.c:3202 (24506):10
Nov 8 17:57:11 unknown com.apple.launchd[1] <Notice>: (UIKitApplication:paradigm.PaymentSystemiPad[0x6c80]) Working around 5020256. Assuming the job crashed.
Nov 8 17:57:11 unknown com.apple.launchd[1] <Warning>: (UIKitApplication:paradigm.PaymentSystemiPad[0x6c80]) Job appears to have crashed: Segmentation fault: 11
Nov 8 17:57:11 unknown SpringBoard[92] <Warning>: Application 'Mail' exited abnormally with signal 9: Killed: 9
Nov 8 17:57:12 unknown DTMobileIS[6377] <Warning>: _memoryNotification : {
OSMemoryNotificationLevel = 0;
timestamp = "2011-11-08 12:27:11 +0000";
}
Nov 8 17:57:12 unknown DTMobileIS[6377] <Warning>: _memoryNotification : <NSThread: 0x1c5361d0>{name = (null), num = 1}
Nov 8 17:57:12 unknown DTMobileIS[6377] <Warning>: _memoryNotification : {
OSMemoryNotificationLevel = 0;
timestamp = "2011-11-08 12:27:12 +0000";
}
Nov 8 17:57:12 unknown SpringBoard[92] <Warning>: Application 'PaymentSystemiPad' exited abnormally with signal 11: Segmentation fault: 11
Nov 8 17:57:12 unknown SpringBoard[92] <Warning>: Application 'iPod' exited abnormally with signal 9: Killed: 9
Nov 8 17:57:12 unknown kernel[0] <Debug>: launchd[6486] Builtin profile: MobileMail (sandbox)
Nov 8 17:57:13 unknown ReportCrash[6485] <Error>: libMobileGestalt loadBasebandMobileEquipmentInfo: CommCenter error: 1:45
Nov 8 17:57:13 unknown ReportCrash[6485] <Error>: libMobileGestalt copyInternationalMobileEquipmentIdentity: Could not get mobile equipment info dictionary
Nov 8 17:57:13 unknown ReportCrash[6485] <Error>: Saved crashreport to /Library/Logs/CrashReporter/LowMemory-2011-11-08-175713.plist using uid: 0 gid: 0, synthetic_euid: 0 egid: 0
Nov 8 17:59:12 unknown SpringBoard[92] <Notice>: MultitouchHID(1f55bf50) uilock state: 0 -> 1
This mean that you have allocated a lot of memory and Received memory warning. I assume you have memory leaks or load many images/resources into memory. Try to research your code and find places where you load resources.