I am using Raspberry Pi Compute Module 3 with Raspbian Buster OS. I have connected Quectel EC25 modem and trying to connect to internet using Network Manager and Modem Manager tool.
Whenever I use new Airtel or Vodafone SIM cards, either Couldn't connect bearer: 'QMI protocol error (14): 'CallFailed'' or couldn't start network: QMI protocol error (79): 'PolicyMismatch' error occurs. After random hours (max. 50 hours), this error gets resolve and connection is being established successfully. I am not able to figure out what exactly the issue is.
List of error as per logs:
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: [/dev/cdc-wdm0] Registered 'wds' (version 1.67) client with ID '22'
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: <info> error: couldn't start network: QMI protocol error (79): 'PolicyMismatch'
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: [/dev/cdc-wdm0] Allocating new client ID...
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: [/dev/cdc-wdm0] Registered 'wds' (version 1.67) client with ID '23'
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: <info> error: couldn't start network: QMI protocol error (14): 'CallFailed'
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: <info> call end reason (1): 'generic-unspecified'
Jan 4 15:43:52 24Seven-78 ModemManager[6574]: <info> verbose call end reason (2,237): [internal] (null)
Jan 4 16:46:54 24Seven-78 ModemManager[6574]: <info> Modem /org/freedesktop/ModemManager1/Modem/1: state changed (registered -> connecting)
Jan 4 16:46:54 24Seven-78 NetworkManager[535]: <info> [1609759014.9170] modem["cdc-wdm0"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Jan 4 16:46:55 24Seven-78 ModemManager[6574]: <info> error: couldn't start network: QMI protocol error (14): 'CallFailed'
Jan 4 16:46:55 24Seven-78 ModemManager[6574]: <info> call end reason (1): 'generic-unspecified'
Jan 4 16:46:55 24Seven-78 ModemManager[6574]: <info> verbose call end reason (6,55): [3gpp] multiple-connection-to-same-pdn-not-allowed
Jan 4 16:46:55 24Seven-78 ModemManager[6574]: [/dev/cdc-wdm0] Allocating new client ID...
QMI Protocol Error 2,237 means ip-support-mismatch and 6,55 means multiple-connection-to-same-pdn-not-allowed. I am not able to understand these and how to resolve them.
Following is the Network Manager connection details:
[connection]
id=MobileConnection
uuid=67bfdda8-e92d-4407-880d-eaf3294a5180
type=gsm
autoconnect-priority=3
interface-name=cdc-wdm0
permissions=
[gsm]
apn=www
number=*99#
[ipv4]
dns-search=
method=auto
route-metric=50
[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto
I am attaching debug logs of Modem Manager.
Logs
Related
Problem
I have a Rails 7 app deployed on render.com, and it doesn't get a lot of traffic (maybe once per day). However, when a few requests do come in, everything seems to running fine for a moment until Puma seems to barf. The incoming requests are from Twilio for a voice call, and the call eventually errors with "We're sorry, an application error has occurred. Goodbye". It seems like something about a "timed out" worker happens, then the worker boots, and whammo! a flood of "Completed 2XX OK" and "Kredis Connected to shared" lines come crashing through like they've been pent up the entire time. THEN, nearly a day later without any outside requests coming in, several log lines about Out-of-sync worker list, no 78 worker come through. My Puma config file is unchanged from what ships with Rails.
Questions
Where might I go look for the offending code? What tools could help me decipher why a Puma worker is timing out? Could it have something to do with how I'm using Redis via Kredis in my app?
Workaround
To get around this issue, I've started to occasionally redeploy my latest commit and that seems to help. I'm not certain, but it seems like inactivity causes Puma to become discombobulated.
Log output
Here's what the offending lines in my log file look like:
... a few requests that complete 200 OK ...
Sep 13 05:53:15 PM [70] ! Terminating timed out worker (worker failed to check in within 60 seconds): 90
... a couple more normal log lines and then ...
Sep 13 05:53:16 PM [70] - Worker 3 (PID: 134) booted in 0.04s, phase: 0
... some more normal log lines and then ...
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.593713 #74] INFO -- : [595ad8e5-fa3a-45a3-8c5b-a506e6c94b69] Completed 204 No Content in 110ms (Allocations: 13681)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.425579 #86] INFO -- : [f1a64c71-8048-4032-8bf6-2e68aa1fa7ba] Completed 204 No Content in 2ms (Allocations: 541)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.595408 #86] INFO -- : [68d19bd9-2286-4f75-a982-5fa3e864d6ac] Completed 200 OK in 105ms (Views: 0.2ms | Allocations: 1592)
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.614951 #76] INFO -- : [e883350f-9a26-4d3d-8f1c-4853285aa71a] Kredis (10.6ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.615787 #76] INFO -- : [fbcd8730-1514-4af5-9332-0bdf0c89fc2d] Kredis (17.2ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.705926 #86] INFO -- : [1f67a177-38f2-4bf5-bd03-1c59a3edb3a4] Kredis (224.1ms) Connected to shared
Sep 13 05:53:16 PM I, [2022-09-13T22:53:16.958386 #76] INFO -- : [e883350f-9a26-4d3d-8f1c-4853285aa71a] Completed 200 OK in 472ms (ActiveRecord: 213.1ms | Allocations: 32402)
Sep 13 05:53:17 PM I, [2022-09-13T22:53:17.034211 #86] INFO -- : [1f67a177-38f2-4bf5-bd03-1c59a3edb3a4] Completed 200 OK in 606ms (ActiveRecord: 256.6ms | Allocations: 17832)
Sep 13 05:53:17 PM I, [2022-09-13T22:53:17.136231 #76] INFO -- : [fbcd8730-1514-4af5-9332-0bdf0c89fc2d] Completed 200 OK in 654ms (ActiveRecord: 88.0ms | Allocations: 37385)
... literally a day later without any other activity ...
Sep 14 05:02:29 AM [69] ! Terminating timed out worker (worker failed to check in within 60 seconds): 78
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] ! Out-of-sync worker list, no 78 worker
Sep 14 05:02:31 AM [69] - Worker 1 (PID: 132) booted in 0.03s, phase: 0
I need to configure the [TRM240 modem][1] to work with a Raspbian system. I followed the procedure reported in the answer [here][2], but the connection isn't working.
I attach some screenshots documenting the procedure and the commands that I issued.
First, look for connected modems:
pi#raspberrypi:~ $ mmcli -L
Found 1 modems:
/org/freedesktop/ModemManager1/Modem/0 [QUALCOMM INCORPORATED] QUECTEL Mobile Broadband Module
Check the unique modem found:
pi#raspberrypi:~ $ mmcli -m 0
/org/freedesktop/ModemManager1/Modem/0 (device id 'e308f44de689f834e8cbc5b2122a4ffca8ddd3f5')
-------------------------
Hardware | manufacturer: 'QUALCOMM INCORPORATED'
| model: 'QUECTEL Mobile Broadband Module'
| revision: 'EC21ECGAR06A04M1G'
| supported: 'gsm-umts
| lte
| gsm-umts, lte'
| current: 'gsm-umts, lte'
| equipment id: '864394040056931'
-------------------------
System | device: '/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2'
| drivers: 'option1, qmi_wwan'
| plugin: 'Generic'
| primary port: 'cdc-wdm0'
| ports: 'ttyUSB0 (qcdm), ttyUSB2 (at), cdc-wdm0 (qmi), wwan0 (net), ttyUSB3 (at)'
-------------------------
Numbers | own : 'unknown'
-------------------------
Status | lock: 'unknown'
| unlock retries: 'unknown'
| state: 'failed'
| failed reason: 'sim-missing'
| power state: 'unknown'
| access tech: 'unknown'
| signal quality: '0' (cached)
-------------------------
Modes | supported: 'allowed: 2g, 3g, 4g; preferred: none'
| current: 'allowed: any; preferred: none'
-------------------------
Bands | supported: 'unknown'
| current: 'unknown'
-------------------------
IP | supported: 'ipv4, ipv6, ipv4v6'
-------------------------
SIM | path: 'none'
-------------------------
Bearers | paths: 'none'
Here, the Status section leaves me a little bit puzzled, but I try to go on.
I tried the following command using different options for ifname: cdc-wdm0, ttyUSB2 and ttyUSB3. They all end up to the same result.
pi#raspberrypi:~ $ sudo nmcli c add type gsm ifname cdc-wdm0 con-name modem apn mobile.vodafone.it
Connection 'modem' (9c34027f-5df5-4fd6-9792-223e225dfd41) successfully added.
Check the connections and try to take up modem:
pi#raspberrypi:~ $ nmcli con
NAME UUID TYPE DEVICE
br-12ff13c2c4e0 26450e11-6671-47c2-91af-6bf96405921b bridge br-12ff13c2c4e0
br-acc4bdcbe6e7 440cfcc0-5318-4a9e-9259-3fb8700cf240 bridge br-acc4bdcbe6e7
docker0 8af67071-3cb2-4235-882f-1cc1f50fa7ca bridge docker0
modem 9c34027f-5df5-4fd6-9792-223e225dfd41 gsm --
Try to take up:
pi#raspberrypi:~ $ nmcli con up modem
Error: Connection activation failed: No suitable device found for this connection.
Status of service NetworkManager:
pi#raspberrypi:~ $ systemctl status NetworkManager
● NetworkManager.service - Network Manager
Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-08-21 07:46:57 UTC; 2h 35min ago
Docs: man:NetworkManager(8)
Main PID: 710 (NetworkManager)
Tasks: 3 (limit: 4915)
Memory: 13.8M
CPU: 3.862s
CGroup: /system.slice/NetworkManager.service
└─710 /usr/sbin/NetworkManager --no-daemon
Aug 21 07:48:53 raspberrypi NetworkManager[710]: <info> [1597996133.0883] device (veth6ffb100): link connected
Aug 21 08:14:11 raspberrypi NetworkManager[710]: <error> [1597997651.4341] audit: failed to open auditd socket: Protocol not supported
Aug 21 08:14:11 raspberrypi NetworkManager[710]: <info> [1597997651.4343] audit: op="device-managed" arg="managed:1" pid=11841 uid=0 result="success"
Aug 21 08:16:48 raspberrypi NetworkManager[710]: <info> [1597997808.9388] audit: op="device-autoconnect" arg="autoconnect:1" pid=12910 uid=1000 result="fail" reason="Not authorized to perform this operat
Aug 21 08:17:41 raspberrypi NetworkManager[710]: <info> [1597997861.0575] audit: op="connection-add-activate" pid=13286 uid=1000 result="fail" reason="gsm: GSM mobile broadband connection requires a 'gsm
Aug 21 08:25:25 raspberrypi NetworkManager[710]: <info> [1597998325.2594] audit: op="connection-add-activate" pid=16224 uid=1000 result="fail" reason="gsm: GSM mobile broadband connection requires a 'gsm
Aug 21 10:06:31 raspberrypi NetworkManager[710]: <info> [1598004391.9223] audit: op="connection-activate" uuid="06583333-e03e-431f-9376-94c38b6b607d" name="modem" result="fail" reason="No suitable device
Aug 21 10:19:24 raspberrypi NetworkManager[710]: <info> [1598005164.3349] keyfile: add connection /etc/NetworkManager/system-connections/modem-9c34027f-5df5-4fd6-9792-223e225dfd41 (9c34027f-5df5-4fd6-979
Aug 21 10:19:24 raspberrypi NetworkManager[710]: <info> [1598005164.3386] audit: op="connection-add" uuid="9c34027f-5df5-4fd6-9792-223e225dfd41" name="modem" pid=27261 uid=0 result="success"
Aug 21 10:20:29 raspberrypi NetworkManager[710]: <info> [1598005229.6409] audit: op="connection-activate" uuid="06583333-e03e-431f-9376-94c38b6b607d" name="modem" result="fail" reason="No suitable device
lines 1-21/21 (END)
Here I see the error <error> [1597997651.4341] audit: failed to open auditd socket: Protocol not supported...
Status of service ModemManager (there are warnings, but not errors):
pi#raspberrypi:~ $ systemctl status ModemManager
● ModemManager.service - Modem Manager
Loaded: loaded (/lib/systemd/system/ModemManager.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-08-21 07:46:57 UTC; 2h 35min ago
Main PID: 712 (ModemManager)
Tasks: 5 (limit: 4915)
Memory: 8.7M
CPU: 4.123s
CGroup: /system.slice/ModemManager.service
├─712 /usr/sbin/ModemManager
└─884 /usr/lib/libqmi/qmi-proxy
Aug 21 07:47:19 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'nas' (version 1.25) client with ID '2'
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Allocating new client ID...
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'wms' (version 1.10) client with ID '1'
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Allocating new client ID...
Aug 21 07:47:20 raspberrypi ModemManager[712]: [/dev/cdc-wdm0] Registered 'uim' (version 1.46) client with ID '1'
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> (ttyUSB2): port attributes not fully set
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> couldn't load Supported Bands: 'QMI operation failed: Cannot send message: QMI service 'dms' version '1.1' required, got version '1.0''
Aug 21 07:47:20 raspberrypi ModemManager[712]: <warn> couldn't load Power State: 'QMI operation failed: Cannot send message: QMI service 'dms' version '1.1' required, got version '1.0''
Aug 21 07:47:55 raspberrypi ModemManager[712]: <warn> Modem couldn't be initialized: Couldn't check unlock status: Couldn't get SIM lock status after 6 retries
Aug 21 07:47:55 raspberrypi ModemManager[712]: <info> Modem: state changed (unknown -> failed)
System info:
pi#raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.66-v7+ #1253 SMP Thu Aug 15 11:49:46 BST 2019 armv7l GNU/Linux
pi#raspberrypi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 9.13 (stretch)
Release: 9.13
Codename: stretch
It may be that the SIM is not working properly or the modem is not recognizing it.
This is what the output
-------------------------
Status | lock: 'unknown'
| unlock retries: 'unknown'
| state: 'failed'
| failed reason: 'sim-missing'
| power state: 'unknown'
| access tech: 'unknown'
| signal quality: '0' (cached)
-------------------------
points out.
Check this answer.
(I didn't find it before posting my question.)
I'm using an Arduino Nano, previously an UNO, to collect data from sensors and then send the data back over a USB connection through a powered hub to a Raspberry Pi model B running Jessie 4.1.19+ Mar, 15 2016 and Python 2.7. The python code uses pyserial readline to get the data.
In general it works fine but occasionally (typically a couple of times a day) it fails with the above message. It seemed the only way to fix it was to unplug the USB connection, reconnect it and restart the program. I altered the code to catch the exception but it wouldn't reopen the connection, giving the error:
serial.serialutil.SerialException: could not open port /dev/ttyUSB0: [Errno 2] No such file or directory: '/dev/ttyUSB0'
This problem occurs with both the UNO (on /dev/ttyAMA0) and the NANO (on /dev/ttyUSB0).
Just recently I discovered that following this fault the Nano had moved to /dev/ttyUSB1. The python code will then run successfully against /dev/ttyUSB1 but when the problem occurs again it switches back to /dev/ttyUSB0.
As a result, I have changed the code to intercept the exception and open the alternative device and carry on. So far this is working fine but its clearly not right.
Can anyone suggest where the problem might lie and how I could go about diagnosing it? I'm not super-technical! Also, I've seen a couple of other posts on this subject but reverting back to a previous kernel level is not really practical in this instance.
Updated to include log output below. There are two devices: NANO (1a86:7523) and an OWL power monitor (0fde:ca05).
$ dmesg -T
[Sat Feb 11 14:53:02 2017] usb 1-1.3-port2: disabled by hub (EMI?), re-enabling...
[Sat Feb 11 14:53:02 2017] usb 1-1.3.2: USB disconnect, device number 71
[Sat Feb 11 14:53:03 2017] dwc_otg_hcd_handle_hc_fsm: 173 callbacks suppressed
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1649 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1650 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] ch341-uart ttyUSB1: usb_serial_generic_read_bulk_callback - urb stopped: -32
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1651 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1652 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1653 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] ch341-uart ttyUSB1: usb_serial_generic_read_bulk_callback - urb stopped: -32
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1654 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1655 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1656 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1657 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] Transfer to device 72 endpoint 0x1 frame 1658 failed - FIQ reported NYET. Data may have been lost.
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: new full-speed USB device number 73 using dwc_otg
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: New USB device found, idVendor=0fde, idProduct=ca05
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: Product: OWL Wireless Electricity Monitor USB version is connected
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: Manufacturer: Silicon Labs
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: SerialNumber: 001A3C26
[Sat Feb 11 14:53:03 2017] cp210x 1-1.3.2:1.0: cp210x converter detected
[Sat Feb 11 14:53:03 2017] usb 1-1.3.2: cp210x converter now attached to ttyUSB0
[Sat Feb 11 14:53:03 2017] usb 1-1.3-port3: disabled by hub (EMI?), re-enabling...
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: USB disconnect, device number 72
[Sat Feb 11 14:53:03 2017] ch341-uart ttyUSB1: ch341-uart converter now disconnected from ttyUSB1
[Sat Feb 11 14:53:03 2017] ch341 1-1.3.3:1.0: device disconnected
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: new full-speed USB device number 74 using dwc_otg
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: New USB device found, idVendor=1a86, idProduct=7523
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: Product: USB2.0-Serial
[Sat Feb 11 14:53:03 2017] ch341 1-1.3.3:1.0: ch341-uart converter detected
[Sat Feb 11 14:53:03 2017] usb 1-1.3.3: ch341-uart converter now attached to ttyUSB2
[Sat Feb 11 14:53:05 2017] cp210x ttyUSB0: cp210x converter now disconnected from ttyUSB0
[Sat Feb 11 14:53:05 2017] cp210x 1-1.3.2:1.0: device disconnected
I know this is an old thread but I hit this issue recently with my Raspberry Pi 3 Model B Rev 1.2 running a stock Debian kernel 5.4.0-3-arm64 while connecting a 3018 CNC Pro controller board. When I lsusb, the board appears as
Bus 001 Device 011: ID 1a86:7523 QinHeng Electronics HL-340 USB-Serial adapter
Whenever I would open the serial port, the usb device would reset. I searched far and wide, applied a bunch of config parameters from different forums but the issue turned out to be power-related for me.
While the 3018 CNC Pro board has an independent power supply, the usb-serial chip on the board is bus-powered (noticed this in lsusb -v). My Raspberry Pi has plenty of available power but I was running the 1m usb cable through a metal clip and along a metal channel which must have had enough inductance to resist the required surge of power through the usb cable. Removing the clip solved the problem.
Hopefully this note helps someone!
Yesterday service worked fine. But today when i checked service's state i saw:
Mar 11 14:03:16 coreos-1 systemd[1]: scheduler.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 11 14:03:16 coreos-1 systemd[1]: Unit scheduler.service entered failed state.
Mar 11 14:03:16 coreos-1 systemd[1]: scheduler.service failed.
Mar 11 14:03:16 coreos-1 systemd[1]: Starting Kubernetes Scheduler...
Mar 11 14:03:16 coreos-1 systemd[1]: Started Kubernetes Scheduler.
Mar 11 14:08:16 coreos-1 kube-scheduler[4659]: E0311 14:08:16.808349 4659 reflector.go:118] watch of *api.Service ended with error: very short watch
Mar 11 14:08:16 coreos-1 kube-scheduler[4659]: E0311 14:08:16.811434 4659 reflector.go:118] watch of *api.Pod ended with error: unexpected end of JSON input
Mar 11 14:08:16 coreos-1 kube-scheduler[4659]: E0311 14:08:16.847595 4659 reflector.go:118] watch of *api.Pod ended with error: unexpected end of JSON input
It's really confused 'cause etcd, flannel and apiserver work fine.
Only some strange logs are for etcd:
Mar 11 20:22:21 coreos-1 etcd[472]: [etcd] Mar 11 20:22:21.572 INFO | aba44aa0670b4b2e8437c03a0286d779: warning: heartbeat time out peer="6f4934635b6b4291bf29763add9bf4c7" missed=1 backoff="2s"
Mar 11 20:22:48 coreos-1 etcd[472]: [etcd] Mar 11 20:22:48.269 INFO | aba44aa0670b4b2e8437c03a0286d779: warning: heartbeat time out peer="6f4934635b6b4291bf29763add9bf4c7" missed=1 backoff="2s"
Mar 11 20:48:12 coreos-1 etcd[472]: [etcd] Mar 11 20:48:12.070 INFO | aba44aa0670b4b2e8437c03a0286d779: warning: heartbeat time out peer="6f4934635b6b4291bf29763add9bf4c7" missed=1 backoff="2s"
So, I'm really stuck and don't know what's wrong. How can i resolve this problem? Or, how can i check details log for scheduler.
journalctl give me same logs like systemd status
Please see: https://github.com/GoogleCloudPlatform/kubernetes/issues/5311
It means apiserver accepted the watch request but then immediately terminated the connection.
If you see it occasionally, it implies a transient error and is not alarming. If you see it repeatedly, it implies that apiserver (or etcd) is sick.
Is something actually not working for you?
I have one applicaion server having multiple nodes. When I am connecting the application through application server, it is working fine. But when I am connecting the application through webserver, it is not working for one node and for all other nodes it works.
What would be the possible cause. I have re-configured the plugin for that particular node many a times. no luck.
I see below logs in plugin log (date may change):
*[Mon Aug 10 11:53:50 2009] 00001dec 00001e04 - ERROR: ws_common: websphereGetStream: Failed to connect to app server on host 'windowsinternal.pt', OS err=78
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ws_common: websphereGetStream: Failed to connect to app server on host 'windowsinternal.pt', OS err=78
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ws_common: websphereExecute: Failed to create the stream
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'windowsinternalNode01_server1'on host 'windowsinternal.pt'; will try another one
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ESI: getResponse: failed to get response: rc = 2
[Mon Aug 10 11:53:51 2009] 00001dec 00001e04 - ERROR: ws_common: websphereHandleRequest: Failed to handle request
[Mon Aug 10 11:53:52 2009] 00001dec 000015b4 - ERROR: ws_common: websphereGetStream: Failed to connect to app server on host 'windowsinternal.pt', OS err=78*
this issue was resolved. There was firewall issue with the default secure port. I just changed the port and it worked.