Google Sign-In: "Certificate for key id xxxx not found" when using google-auth Python package - jwt

Question

I'm maintaining a website and its mobile apps (iOS and Android). For Google Sign-In in mobile app, I'm using google-auth Python package on the server side.
Since about one month ago, I started to receive error reports related to Google Sign-In from the server side. The error message looks like the following:
Certificate for key id 728f4016652079b9ed99861bb09bafc5a45baa86 not found.
The server-side Google Sign-In authentication backend follows this document:
from google.oauth2 import id_token
from google.auth.transport import requests
# ...
try:
# The following line may raise ValueError with message:
# Certificate for key id xxxx not found.
id_info = id_token.verify_oauth2_token(google_id_token, requests.Request())
if id_info['aud'] not in VALID_CLIENT_IDS:
logger.error('Invalid aud from Google ID token: %s', id_info['aud'])
raise ValueError('Unverified audience.')
# ...
except ValueError as exc:
logger.error('Fail to verify Google ID token: %s', exc, extra={'request': request})
The error comes from the google.auth.jwt module, when verifying the Google-issued JWT against a list of Google public certificates.
Diving into the google-auth code, I can see that verify_oauth2_token() function is fetching Google public certificates from the URL https://www.googleapis.com/oauth2/v1/certs. It seems that sometimes, for some Google ID token sent from some Android phones, the key id cannot be found in that URL.
Here are some other details which might be useful:
It seems that iOS app does not have this kind of issue. From the USER_AGENT header, I can see that the error happens only in Android app (USER_AGENT=okhttp/3.11.0). And it happens only in some Android device, not all.
I was wondering if this happens only for Android phones from China (e.g. if they are connecting via a VPN). So I also checked the user IP address. But it turned out that some of those users were from Europe.
Some key id is recurring again and again in the server error logs. For example, the key id aa436c3f63b281ce0d976da0b51a34860ff960eb is seen dozens of times, from the beginning of November to now (end of Dec).
I am constantly seeing this error, several times (10 ~ 30 times) every day.
My website is running on the following environment:
OS: Linux (CentOS 7) 64-bit
Apache 2 with mod_wsgi 4.5.24
Python 3.6.7 and Django 2.1.2
google-auth version: tried both 1.3.0 and 1.6.1
As I could not reproduce this issue either with my iPhone, or with my Android phone (Huawei P20, bought in France), I am completely stuck.
But, one of my friends is having this issue now, and he bought his Android phone from Hongkong. This makes me think if it is possible that for some countries, Google Sign-In uses some different certs other than the public ones at https://www.googleapis.com/oauth2/v1/certs?
I don't think this is a bug in google-auth package. I'm wondering if any of you have ever heard of this error, and could give me a hint about the possible cause of it?
Thanks in advance!

Answer 1

OK, I finally figured it out. I'm posting my findings here, hoping that it could help someone else.
The server-side Python code has nothing wrong. The reason why it failed was that, the client app submitted an expired Google ID token.
Here's the fixed version of my LoginActivity:
...
#OnClick(R.id.google_sign_in_button)
void loginWithGoogle() {
//
// If user has already signed in to our app with Google, sign him out first.
//
// NOTE: This step is required, or the ID token might not pass the server-side validation!
//
// After sign-in, we need to get the user's ID token issued and signed by Google, and send
// it back to our server for validation.
//
// Google is rotating its OAuth2 certificate regularly, so an old ID token issued long time
// ago by Google might not pass the server-side validation -- if the certificate used to
// sign the ID token has expired.
//
// This may happen when user has already signed in to our app with Google. In such case,
// the ID token we get from the user's Google account is obsolete. Our server will fail to
// validate it, with the error message:
//
// Fail to verify Google ID token: Certificate for key id xxx not found.
//
// Google recommends using the `silentSignIn` method for the already-signed-in user
// (see step 2 of: https://developers.google.com/identity/sign-in/android/backend-auth).
// For the sake of simplicity, we don't do that. Instead, we go directly to step 3
// by signing user out, giving him the option to sign-in again.
//
final GoogleSignInAccount account = GoogleSignIn.getLastSignedInAccount(this);
if (account != null) {
// User has already signed in: Sign out and sign in again.
// NOTE: THIS IS THE FIX TO MY PROBLEM.
mGoogleSignInClient.signOut().addOnCompleteListener(this, new OnCompleteListener<Void>() {
#Override
public void onComplete(#NonNull Task<Void> task) {
startGoogleSignInActivity();
}
});
} else {
// User is not yet signed in: Start the Google sign-in flow.
startGoogleSignInActivity();
}
}
private void startGoogleSignInActivity() {
final Intent intent = mGoogleSignInClient.getSignInIntent();
startActivityForResult(intent, REQUEST_LOGIN_WITH_GOOGLE);
}
The key point is that: I need to check if user has already signed in with Google. If yes, I need to sign the user out and re-start the Sign-in-with-Google activity.
As Android has native support for Google account, I suppose the OS may cache user's Google account if user is already authenticated (in some other app or system-wide). But that cached account may probably contain an expired ID token. Forcing to sign the user out and re-sign-in will give me a fresh new ID token.
This also explains why my iOS app does not have this issue. Because iOS never caches user's Google account.