How to set up VPC network peering from multiple App Engine projects to Mongo Atlas - mongodb

Question

I have an App Engine app, which connects securely to Mongo Atlas via a network peering connection which is all working fine.
I have come to want to make the app multi-region, which means creating multiple projects and therefore reproducing the various GCP infrastructure, including the peering connection. However when reproducing this connection, I cannot due to the IP conflict at the Mongo Atlas side between the two "default" VPC in each project.
I can create the VPC network peering in the GCP end OK, sharing the "default" VPC and setting the same Mongo project/network IDs. The default VPC has ranges for each region , e.g. us-west1=10.138.0.0/20, us-west2=10.168.0.0/20 (my original app region), and us-west4=10.182.0.0/20 - the 2nd region I am setting up.
At the Mongo DB end, their CIDR block is fixed at 192.168.0.0/16 and cannot be changed. But when I enter the new GCP project ID and "default" VPC, it throws this error:
Error trying to process asynchronous operation: An IP range in the peer network (10.138.0.0/20) overlaps with an IP range (10.138.0.0/20) in an active peer (peer-ABCXYZ) of the local network.
I understand that the IP ranges can't overlap as there would be routing ambiguity. So I'd like to know how to resolve this and connect from both projects.
I noticed that the error was about 10.138 which is us-west1 region, which I'm not even using. So is there a way to limit each VPC peering to only share the region for the project? If I could do that for each, there would be no overlap.
Mongo DB has a document about this problem, but this only discusses an AWS solution and only from their perspective, not saying how to set up the other end.
https://docs.atlas.mongodb.com/security-vpc-peering/#network-peering-between-an-service-vpc-and-two-virtual-networks-with-identical-cidr-blocks
GCP has a document about the problem, but doesn't seem to offer a resolution, just "you can't do this"
https://cloud.google.com/vpc/docs/vpc-peering#overlapping_subnets_at_time_of_peering
I'm guessing I will need to create a new VPC perhaps with region-limited subnets and only share that VPC? I had a look at "Create VPC network" but it got complex pretty quickly.
What I want is something like:
Project A, us-west2=10.168.0.0/20 <==> Mongo Atlas 192.168.0.0/16
Project B, us-west4=10.182.0.0/20 <==> Mongo Atlas 192.168.0.0/16
This question is similar, but there is no specific instructions (as the OP didn't want the second connection anyway) Mongodb Atlas Google Cloud peering fails with an ip range in the local network overlaps with an ip range in an active peer
Update
I have since found one of the reasons this became a problem is because when originally setting up the first app 2 years ago, I just used the "default" VPC which itself defaults to "auto mode" which automatically creates subnets for all regions present and future. This can be a time-saver, but GCP recommends not to use this in production - for many reasons including my problem! If you want more control over the subnets and avoiding conflicts etc, they recommend you use a "custom mode" VPC where you have to define all the subnets yourself.
In my case I didn't need this super VPC of all possible regions in the world, but just one region. So now I will have to convert it to custom-mode and prune back the other regions I'm not using in this project, to be able to resolve the overlap (even if I do use a single-region subnet in another project, I still need to remove them from the original project to avoid the conflict).

Answer 1

You are right, if you use default VPC, you have VPCs in all regions and the peering failed because of the overlap.
There is 2 solutions:
Create a custom VPC in each region/project to create a clean peering
Or (my favorite), create a shared VPC and add all the region/project to the host project. At the end, it's the same project, but only in multiregion, sharing the VPC layer make a lot of sense.

Answer 2

Guillaume's answer is correct, but I thought I'd add my specific working recipe including how I avoided the conflict without having to reconfigure my original app.
I was going to convert my original app's auto-mode VPC into a custom one, then remove the regions I'm not using (all but us-west2). I practiced this in a different project and seemed to work quickly and easily, but I wanted to avoid any disruption to my production app.
After researching the IP ranges used by the auto-mode VPC, I realised I can just create a new VPC in my second region using any spare "local" IP range, as long as I avoid both the GCP auto-range of 10.128.0.0/20 (10.128.0.0 - 10.255.255.255) and the Mongo Atlas range of 192.168.0.0/16 (192.168.0.0 - 192.168.255.255), so I chose 10.1.0.0/16.
Then performed these steps:
Create custom VPC "my-app-engine" in my second region project
Add 1 subnet "my-app-engine-us-west4" region: "us-west4" 10.1.0.0/16
Add VPC Peering to this network both at GCP + Mongo Atlas and wait for it to connect
Add the subnet range 10.1.0.0/16 to Atlas Network Access > IP Access List
Re-deployed the app into this VPC with extra app.yaml settings:
network: my-app-engine
subnetwork_name: my-app-engine-us-west4
You have to specify subnetwork_name as well in the app.yaml for custom VPCs, but not for auto-mode ones.
The "IP Access List" caught me out for a while as I'd forgotten you also have to open the Mongo firewall to the new VPC Peering, and was still getting connection timeouts to the cluster, even though the peering was setup.
So now I have two VPC peerings, the original overstuffed one, and this new slim one on a custom network. I will eventually redeploy the old app in this slimmer pattern once the new region is working.