Github action cache image used multi-stage build - github

Question

I have a multi-stage build that uses a docker image which contains all the node_modules needed for a build and other dependencies. I used that in a multi-stage docker build to build a production dist for a project. In the 2nd stage I use a much smaller docker to store the dist files so it's easier to distribute.
Dockerfile for the build:
FROM ghcr.io/dancing-star/packages-image-full:latest AS builder
# Make the dir
WORKDIR /var/dancing-star
COPY . .
# Set the cache location
RUN npm config set cache /tmp/node-cache --global \
&& npm run prod \
&& rm -rf src \
&& ls -l
# Dist stage
FROM alpine
WORKDIR /var/dancing-star
COPY --from=builder ["/var/dancing-star/dist", "/var/dancing-star/assets", "/var/dancing-star/config", "/var/dancing-star/scripts", "/var/dancing-star/*.json", "/var/dancing-star/*.md", "/var/dancing-star/*.js", "/var/dancing-star/"]
RUN ls -l
#CMD [ "pm2-runtime", "start", "app.pm2.json", "--env", "production" ]
The ghcr.io/dancing-star/packages-image-full contains the node_modules and works fine. I used a github action to cache ghcr.io/dancing-star/packages-image-full but it doesn't seem to be caching it very well since it's constantly pulling it again.
.github/workflows/docky.yaml
name: Build Docker
on:
push:
branches: [ main ]
tags:
- v*
env:
IMAGE_NAME: project-docker
BUILD_IMAGE_NAME: ghcr.io/naologic/dancing-star/packages-image-full
BUILD_IMAGE_PATH: ghcr.io/naologic/dancing-star/packages-image-full:latest
jobs:
build:
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Checkout
uses: actions/checkout#v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action#v2
- name: Login to GitHub Container Registry
uses: docker/login-action#v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.DOCKER_DEPLOY_GITHUB }}
- name: Build and push
uses: docker/build-push-action#v4
with:
context: .
file: Dockerfile
push: true
tags: ${{ env.BUILD_IMAGE_PATH }}
cache-from: type=registry,ref=${{ env.BUILD_IMAGE_NAME }}:buildcache
cache-to: type=registry,ref=${{ env.BUILD_IMAGE_NAME }}:buildcache,mode=max
IMPORTANT: ghcr.io/dancing-star/packages-image-full is shared between many builds so ideally it would be a shared cache.
How should I use actions to cache ghcr.io/dancing-star/packages-image-full for the build?
My solution doesn't seem to be working very well.
I tried the current action, cache still pulls for 2 minutes before starting
I tried without cache, build time is about the same