When I run CTS after a few hours the adb connection to device becomes unresponsive - android-source

Question

I am executing CTS on Jacinto 6 Evaluation Module (ti-jacinto6evm) and I'm encountering a number of test case failures that I don't understand.
I started by building both AOSP and CTS. Both builds were just fine. I can flash my test hardware (ti-jacinto6evm) and then I followed the instructions for setting up CTS. I have run CTS for more then 10 times on the same device and every time I got different results. The ti-jacinto6 device randomly gets hanged during execution of the test cases.
Most of the time target gets hanged and it show following error:
Reason: 'Failed to receive adb shell test output within 600000 ms. Test may have timed out, or adb connection to device became unresponsive'. Check device logcat for details
Device 170090035a700002 shell is unresponsive
05-30 04:52:21 W/TestInvocation: Invocation did not complete due to device 170090035a700002 becoming not available. Reason: Could not find device 170090035a700002
on the below test cases my target hangs:
CtsPreference2TestCases
CtsUiHostTestCases
CtsServicesHostTestCases
CtsTrustedVoiceHostTestCases
CtsTransitionTestCases
CtsAppTestCases
CtsGraphicsTestCases
CtsCameraTestCases
CtsWebkitTestCases
CtsFragmentTestCases
CtsViewTestCases
So I just excluded those test cases from the CTS and again ran CTS with the following command:
run cts --skip-preconditions --exclude-filter CtsPreference2TestCases --exclude-filter CtsServicesHostTestCases --exclude-filter CtsUiHostTestCases --exclude-filter CtsTrustedVoiceHostTestCases --exclude-filter CtsAppTestCases --exclude-filter CtsGraphicsTestCases --exclude-filter CtsTransitionTestCases --exclude-filter CtsCameraTestCases --exclude-filter CtsWebkitTestCases --exclude-filter CtsFragmentTestCases --plan cts
Problem 1
I am facing a problem where some test cases are running properly for the first time, but when I run CTS for the second time, they fail some passed test case(s).
1st iteration. On this iteration 166 modules passed:
Testcase name
Passed
Failed
Total executed
armeabi-v7a CtsWebkitTestCases
201
12
213
2nd iteration. On this iteration 91 modules passed:
Testcase name
Passed
Failed
Total executed
armeabi-v7a CtsWebkitTestCases
80
1
81
Problem 2
When CTS gets stuck on some testcases it shows a TimeoutException:
com.android.ddmlib.TimeoutException
at com.android.ddmlib.AdbHelper.read(AdbHelper.java:767)
at com.android.ddmlib.AdbHelper.read(AdbHelper.java:736)
at com.android.ddmlib.AdbHelper.readAdbResponse(AdbHelper.java:222)
at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:456)
at com.android.ddmlib.AdbHelper.executeRemoteCommand(AdbHelper.java:382)
at com.android.ddmlib.Device.executeShellCommand(Device.java:617)
at com.android.tradefed.device.NativeDeviceStateMonitor.waitForDeviceShell(NativeDeviceStateMonitor.java:170)
at com.android.tradefed.device.WaitDeviceRecovery.recoverDevice(WaitDeviceRecovery.java:142)
at com.android.tradefed.device.NativeDevice.recoverDevice(NativeDevice.java:1720)
at com.android.tradefed.device.NativeDevice.performDeviceAction(NativeDevice.java:1661)
at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:615)
at com.android.tradefed.device.NativeDevice.runInstrumentationTests(NativeDevice.java:698)
at com.android.tradefed.testtype.InstrumentationTest.runWithRerun(InstrumentationTest.java:797)
at com.android.tradefed.testtype.InstrumentationTest.doTestRun(InstrumentationTest.java:740)
at com.android.tradefed.testtype.InstrumentationTest.run(InstrumentationTest.java:643)
at com.android.tradefed.testtype.AndroidJUnitTest.run(AndroidJUnitTest.java:233)
at com.android.compatibility.common.tradefed.testtype.ModuleDef.run(ModuleDef.java:250)
at com.android.compatibility.common.tradefed.testtype.CompatibilityTest.run(CompatibilityTest.java:506)
at com.android.tradefed.invoker.TestInvocation.runTests(TestInvocation.java:761)
at com.android.tradefed.invoker.TestInvocation.prepareAndRun(TestInvocation.java:446)
at com.android.tradefed.invoker.TestInvocation.performInvocation(TestInvocation.java:300)
at com.android.tradefed.invoker.TestInvocation.invoke(TestInvocation.java:886)
at com.android.tradefed.command.CommandScheduler$InvocationThread.run(CommandScheduler.java:567)
What is the reason behind this failure?

Answer 1

No need to rerun the tests that already past you can continue and run only the tests that failed or were not ran, use command
l r
to get the results and then use the first column number session id of the run to continue test from like this:
run cts --retry 12
where 12 is the run session id displayed in the first column of l r.
adb disconnection indeed can affect the test cases, I use this small script to reconnect, you can modify it to suit your needs.
cat adb_retry.sh:
while :
do
if ((`adb devices | wc -l` < 3 )); then
echo Connection for $1 droped out
echo retrying
adb connect "$1"
fi
sleep 5
echo Watching...
done