Google Cloud PostgreSQL: Utilization remains at 100% - postgresql

Question

I am using Google Cloud PostgreSQL which has utilization CPU 100%. I have upgraded the instance to use 2 cores. Now the instance is running on 2 CPU's and 3.75Gb of RAM. Still the instance is using 100% of CPU resources. Again, I have upgrade the instance to 6 cores and 12Gb of RAM, but still no change in CPU utilization. Here are some stats metrics:
I want any thought about why this is happening, how can I figure out the solution?
I have checked the number of queries running on PostgreSQL. Number of queries is less than 100 and execution time is less than 30 seconds. PostgreSQL verison is 9.6

Answer 1

I've been doing this daily now, I'll share how I debug this problem.
Firsts of all, install extension pgstatstatements so it will store all execution statistics of all SQL statements executed on the server.
After that, it's easy...
This query will show most "expensive" queries:
SELECT substring(query, 1, 50) AS short_query,
round(total_time::numeric, 2) AS total_time,
calls,
round(mean_time::numeric, 2) AS mean,
round(max_time::numeric, 2) AS max_time,
round((100 * total_time / sum(total_time::numeric) OVER ())::numeric, 2) AS percentage_cpu,
query
FROM pg_stat_statements
ORDER BY total_time DESC LIMIT 10
And this one to reset the statistics, useful when you want to debug a specific period:
SELECT pg_stat_statements_reset()
In order to see which queries are running currently on the server:
SELECT user, pid, client_addr, query, query_start, NOW() - query_start AS elapsed
FROM pg_stat_activity
WHERE query != '<IDLE>'
-- AND EXTRACT(EPOCH FROM (NOW() - query_start)) > 1
ORDER BY elapsed DESC;
If you have a better way to debug the performance please tell me!
Also if some GCP engineers are reading this, please enable more metrics that can enable us to trace the problem. Example process CPU on the server can tell which DB/Schema is taking too much CPU.
EDIT:
Google released query insights and it's useful when you don't want to make your hands dirty!
I still use pgstatstatements!