Puppeteer's page.cookies() not retrieving all cookies shown in the Chrome dev tools - google-chrome-devtools

Question

Using puppeteer, I am trying to retrieve all cookies for a specific web site (i.e. https://google.com) from Node.js.
My code is:
// Launch browser and open a new page
const browser = await puppeteer.launch({ headless: true, args: ['--disable-dev-shm-usage'] });
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle2' });
var cookies = await page.cookies();
console.log(cookies);
await browser.close();
It only retrieves 2 cookies, named 1P_JAR and NID. However, when I open the Chrome Dev tools, it shows a lot more.
I tried using the Chrome Dev Tools directly instead of puppeteer but I am getting the same results.
Is there another function I should call? Am I doing it correctly?

Answer 1

The page.cookies() call only gets cookies that are available to JavaScript applications inside the browser, and not the ones marked httpOnly, which you see in the Chrome DevTools. The solution is to ask for all available cookies through the Devtools protocol and then filter for the site you're interested in.
var data = await page._client.send('Network.getAllCookies');

Answer 2

You can utilise Chrome DevTools Protocol -> getAllCookies
To get all browser cookies, regardless of any flags.
const client = await page.target().createCDPSession();
const cookies = (await client.send('Network.getAllCookies')).cookies;
This will also play nice with typescript and tslint since something like
const cookies = await page._client.send('Network.getAllCookies');
Will raise an error TS2341: Property '_client' is private and only accessible within class 'Page'..

Answer 3

Thanks #try-catch-finally. I got it resolved and it was a simple rookie mistake.
I was comparing cookies in my own Google Chrome instance with the Puppeteer instance. However, in my instance, I was logged in to my Google account and Puppeteer (obviously) was not.
Google uses 2 cookies when you are NOT logged in and 12 when you are logged in.

Answer 4

If you use Playwright in place of Puppeteer, httponly cookies are readily accessible:
const { chromium } = require('playwright')
(async () => {
const browser = await chromium.launch()
const context = await browser.newContext()
const page = await context.newPage()
await page.goto('https://google.com', { waitUntil: 'networkidle' })
let allCookies = await context.cookies()
console.log (allCookies)
})();
returns:
[
{
sameSite: 'None',
name: '1P_JAR',
value: '2021-01-27-19',
domain: '.google.com',
path: '/',
expires: 1614369040.389115,
httpOnly: false,
secure: true
},
{
sameSite: 'None',
name: 'NID',
value: '208=VXtmbaUL...',
domain: '.google.com',
path: '/',
expires: 1627588239.572781,
httpOnly: true,
secure: false
}
]

Answer 5

Just use it await page.goto('https://google.com', { waitUntil: 'networkidle2' }). And you can get all the cookies related.