Jenkins Intermittent Hangs on git clone/checkout from github on pipeline jobs - github

Question

Have you seen and do you have a solution for or investigation ideas for this problem:
jenkins pipeline jobs using Pipeline from GitHub SCM intermittently hang on clone/checkout
nothing reported in jenkins logs about this situation
only seems to happen when we have a lot of activity (usually right before sprint demos)
clears after some period of time - haven't been able to detect much a pattern yet
Checking out git git#github.com:MyTeam/myproject into
/var/lib/jenkins/jobs/myproject-dev-deploy/workspace#script to read deploy/dev-Jenkinsfile
git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url git#github.com:MyTeam/myproject # timeout=10
Fetching upstream changes from git#github.com:MyTeam/myproject
git --version # timeout=10
using GIT_SSH to set credentials Deploy key for myproject
git fetch --tags --progress git#github.com:MyTeam/myproject +refs/heads/:refs/remotes/origin/```

Answer 1

A horrible intermittent problem in a java app? To the JVM settings we go. Luckily CloudBees wrote a great article on this Tuning Jenkins GC For Responsiveness and Stability with Large Instances
Turns out this wasn't the prime factor. The real cause was network configuration and impact of a backup job on saturation. Still, an excellent article.
Here's how we debugged it:
Github support gave us this https://github-debug.com/ script and we augmented it thus
`
#!/usr/bin/env bash
echo Clearing Previous Run
rm -rf /tmp/debug-repo-http
rm -rf /tmp/debug-repo-ssh
echo clone1: `date`
time git clone https://github.com/github/debug-repo /tmp/debug-repo-http
echo clone2: `date`
time git clone git#github.com:github/debug-repo /tmp/debug-repo-ssh
echo Ping: `date`
time ping -c 10 github.com
#echo traceroute: `date`
#traceroute github.com
Setup jenkins job to attempt checkout, fail and report if it took longer than 30s - once via pipeline + plugin and once via shell script
Began running various tests: dump all docker images and re-pull, s3 pull backup, s3 push backup etc.
Repeat tests in another network area and compare results

Answer 2

I had meet this issue by git clone command and it's cost me 2 days to resolved this problems.
If you run git command by windows shell/batch on jenkins, it maybe cause by git ssh key issue...
Jenkins windows shell or batch command will use 'system' account to process git command and its ssh key setting is different form your gerrit plugin or login account.
A. Different git ssh key path
The git ssh key of 'system' account is setting in following path:
C:\Windows\System32\config\systemprofile\.ssh
C:\Windows\SysWOW64\config\systemprofile\.ssh
You can check ssh key in these paths for system account.
B. Register key in know_host file
Jenkins will not auto import new key of known host to your know_host file.
You can do it mannually by following steps:
Login your local account in windows
SSH this git remote path directly ssh <git remote web site> -p <port>, type yes to register the key
Copy the registered key form local account know_host (C:\Users\.ssh) you will see the bottom of file likes:
[gerrit.cartography.com]:29168,[10.25.174.78]:29418 ssh-rsa AAAAB3NzaC1yc2EAASSSSAQABAAAAgQDPHbDKCHL3oPbzDn3qaVZR8S1EZx/rb500Th/fjQYg5jrJ7PBMVVUyNTpJ2moje5hLYjNsVtXtcoNEZjImGul2dl3f+eHrCB27X5198jr8Wj66I/xsQCLEngd7l3velsdfgd3432fsdDiFaE9INjDjgngfNzlFA+CYpFsnaorZ6XSPQ==
Paste to 'system' know_host file.
C:\Windows\System32\config\systemprofile\.ssh
C:\Windows\SysWOW64\config\systemprofile\.ssh
I think you will resolve your problem now !
Enjoy it!