ssh permission denied when connecting from Kubernetes pod to remote host - kubernetes

Question

I'm trying to ssh from a pod into a remote server while specifying an identity file. This fails with the following error:
admin#123.123.123.123: Permission denied (publickey).
I've made sure I can connect from my local host with the same set of public and private keys. It only fails when I try to connect from a bash shell inside the pod's container.
My Job definition is as follows :
---
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-volumes-population-{{ .Release.Revision }}
spec:
template:
spec:
containers:
- name: populate-volumes
image: {{ .Values.gitlab.image_repository.repository }}/{{ .Values.phpfpm.image.name }}:{{ .Values.phpfpm.image.version }}
imagePullPolicy: IfNotPresent
ports:
- name: ssh
containerPort: 22
args:
- /bin/bash
- -c
- |
echo "Testing ssh connection..."
ssh -i/etc/ssh/hetzner_box admin#123.123.123.123
volumeMounts:
- name: hetzner-box-identity
mountPath: /etc/ssh/hetzner_box.pub
subPath: .pub
- name: hetzner-box-identity
mountPath: /etc/ssh/hetzner_box
subPath: .key
volumes:
- name: hetzner-box-identity
secret:
secretName: {{ .Release.Name }}-hetzner-box-identity
defaultMode: 256
items:
- key: .pub
path: .pub
- key: .key
path: .key
Edit 1:
After further inquiries I've manage to notice that the key pair is passphrase less. I've managed to login using a different key pair, protected by a passphrase. My goal is automation and is therefore unacceptable to have passphrase protected keys. Is there a reason the ssh daemon is refusing to authenticate a passphrase less key?