Velero + MinIO: Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; - kubernetes

Question

I'm getting this issue below. Anyone has an idea what could be wrong?
user#master-1:~$ kubectl logs -n velero velero-77b544f457-dw4hf
# REMOVED
An error occurred: some backup storage locations are invalid: backup store for location "aws" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
status code: 400, request id: A3Q97JKM6GQRNABA, host id: b6g0on189w6hYgCrId/Xr0BP44pXjZPy2SqK2t7bn/+Ggq9FUY2N3KQHYRcMEuCCHY2L2vfsYEo=; backup store for location "velero" is invalid: rpc error: code = Unknown desc = AuthorizationHeaderMalformed: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'
status code: 400, request id: YF6DRKN7MYSXVBV4, host id: Y8/Gufd7R0BlZCIZqbJPfdAjVqK8+WLfWoANBDnipDkH421/vGt0Ne2E/yZw2bYf7rfms+rGxsg=
user#master-1:~$
I have installed Velero 1.4.2 with Helm chart:
user#master-1:~$ helm search repo velero --versions | grep -e 2.12.17 -e NAME
NAME CHART VERSION APP VERSION DESCRIPTION
vmware-tanzu/velero 2.12.17 1.4.2 A Helm chart for velero
user#master-1:~$
I used this command to install:
helm install velero vmware-tanzu/velero --namespace velero --version 2.12.17 -f velero-values.yaml \
--set-file credentials.secretContents.cloud=/home/era/creds-root.txt \
--set configuration.provider=aws \
--set configuration.backupStorageLocation.name=velero \
--set configuration.backupStorageLocation.bucket="velero" \
--set configuration.backupStorageLocation.prefix="" \
--set configuration.backupStorageLocation.config.region="us-east-1" \
--set image.repository=velero/velero \
--set image.tag=v1.4.2 \
--set image.pullPolicy=IfNotPresent \
--set initContainers[0].name=velero-plugin-for-aws \
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.1.0 \
--set initContainers[0].volumeMounts[0].mountPath=/target \
--set initContainers[0].volumeMounts[0].name=plugins \
--replace
My credential files passed:
$ cat creds-root.txt
[default]
aws_access_key_id=12345678
aws_secret_access_key=12345678
Helm values file:
user#master-1:~$ cat velero-values.yaml
configuration:
provider: aws
backupStorageLocation:
name: minio
provider: aws
# caCert: null
bucket: velero
config:
region: us-east-1
credentials:
useSecret: true
existingSecret: cloud-credentials
secretContents: {}
extraEnvVars: {}
backupsEnabled: true
snapshotsEnabled: true
deployRestic: true
MinIO snapshot resource (MinIO is working at 192.168.2.239:9000):
# For MinIO
---
apiVersion: velero.io/v1
kind: VolumeSnapshotLocation
metadata:
name: minio
namespace: velero
spec:
provider: openebs.io/cstor-blockstore
config:
bucket: velero
prefix: cstor
provider: aws
# The region where the server is located.
region: us-east-1
# profile for credential, if not mentioned then plugin will use profile=default
profile: user1
# Whether to use path-style addressing instead of virtual hosted bucket addressing.
# Set to "true"
s3ForcePathStyle: "true"
# S3 URL, By default it will be generated from "region" and "bucket"
s3Url: http://192.168.2.239:9000
# You can specify the multipart_chunksize here for explicitness.
# multiPartChunkSize can be from 5Mi(5*1024*1024 Bytes) to 5Gi
# For more information: https://docs.min.io/docs/minio-server-limits-per-tenant.html
# If not set then it will be calculated from the file size
multiPartChunkSize: 64Mi
# If MinIO is configured with custom certificate then certificate can be passed to plugin through caCert
# Value of caCert must be base64 encoded
# To encode, execute command: cat ca.crt |base64 -w 0
# caCert: LS0tLS1CRU...tRU5EIENFUlRJRklDQVRFLS0tLS0K
# If you want to disable certificate verification then set insecureSkipTLSVerify to "true"
# By default insecureSkipTLSVerify is set to "false"
insecureSkipTLSVerify: "true"
aws resource which seems failing:
$ k get backupstoragelocation -n velero aws -o yaml
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
annotations:
helm.sh/hook: post-install,post-upgrade
helm.sh/hook-delete-policy: before-hook-creation
creationTimestamp: "2021-04-15T08:23:38Z"
generation: 3
labels:
app.kubernetes.io/instance: velero
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: velero
helm.sh/chart: velero-2.12.17
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:helm.sh/hook: {}
f:helm.sh/hook-delete-policy: {}
f:labels:
.: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/managed-by: {}
f:app.kubernetes.io/name: {}
f:helm.sh/chart: {}
f:spec:
.: {}
f:config:
.: {}
f:region: {}
f:objectStorage:
.: {}
f:prefix: {}
f:provider: {}
manager: Go-http-client
operation: Update
time: "2021-04-15T08:23:38Z"
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
f:objectStorage:
f:bucket: {}
manager: kubectl-edit
operation: Update
time: "2021-04-15T17:52:46Z"
name: aws
namespace: velero
resourceVersion: "1333724"
selfLink: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/aws
uid: a51033b2-e53d-4751-9110-c9649de6aa67
spec:
config:
region: us-east-1
objectStorage:
bucket: velero
prefix: backup
provider: aws
user#master-1:~$
For some reason no plugins are listed:
user#master-1:~$ velero plugin get
user#master-1:~$
Velero is obviously crashing because of original issue:
user#master-1:~$ kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
restic-nqpsl 1/1 Running 0 7m52s
restic-pw897 1/1 Running 0 7m52s
restic-rtwzd 1/1 Running 0 7m52s
velero-77b544f457-dw4hf 0/1 CrashLoopBackOff 5 5m59s
user#master-1:~$
More resources:
user#master-1:~$ k get BackupStorageLocation -n velero
NAME PHASE LAST VALIDATED AGE
aws 10h
velero 11m
user#master-1:~$ k get volumesnapshotlocation -n velero
NAME AGE
default 11m
minio 39h
velero-snapshot 9h
user#master-1:~$
My MinIO service is started using Docker Compose and working fine:
version: '3.8'
services:
minio:
container_name: minio
hostname: minio
build:
context: .
dockerfile: Dockerfile
restart: always
ports:
- "0.0.0.0:9000:9000"
environment:
# ROOT
MINIO_ACCESS_KEY: 12345678
MINIO_SECRET_KEY: 12345678
MINIO_REGION: us-east-1
command: server --address :9000 /data
volumes:
- ./data:/data
Unknown PHASE for backup locations:
user#master-1:~$ velero get backup-locations
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE
aws aws velero/backup Unknown Unknown ReadWrite
velero aws velero Unknown Unknown ReadWrite
user#master-1:~$
Test MinIO access separately:
bash-4.3# AWS_ACCESS_KEY_ID=12345678 AWS_SECRET_ACCESS_KEY=12345678 aws s3api get-bucket-location --endpoint-url http://192.168.2.239:9000 --bucket velero
{
"LocationConstraint": "us-east-1"
}
bash-4.3#
Secrets are correct:
user#master-1:~$ k get secret -n velero cloud-credentials -o yaml | head -n 4
apiVersion: v1
data:
cloud: W2RlZmF-REMOVED
kind: Secret
user#master-1:~$
user#master-1:~$ k get secret -n velero
NAME TYPE DATA AGE
cloud-credentials Opaque 1 91m
default-token-8rwhg kubernetes.io/service-account-token 3 2d20h
sh.helm.release.v1.velero.v1 helm.sh/release.v1 1 45m
velero Opaque 0 2d19h
velero-restic-credentials Opaque 1 40h
velero-server-token-8zm9k kubernetes.io/service-account-token 3 45m
user#master-1:~$

Answer 1

The problem was missing configuration:
--set configuration.backupStorageLocation.config.s3Url="http://192.168.2.239:9000" \
--set configuration.backupStorageLocation.config.s3ForcePathStyle=true \