why my Velero backup fail for minio storage with error An error occurred: gzip: invalid header? - kubernetes

Question

I have install the minio example from velero vmware-tanzu. The minio example setup is running and I have set the nodePort service. Then I follow the following command to install velero.
velero install --provider aws --plugins velero/velero-plu gin-for-aws:v1.0.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=false --backup-location-config r egion=minio,s3ForcePathStyle="true",s3Url=http://123.123.123.123:30804
When I check the logs for velero I see this error.
time="2023-02-05T10:45:30Z" level=error msg="fail to validate backup store" backup-storage-location=velero/default controller =backup-storage-location error="rpc error: code = Unknown desc = InvalidArgument: S3 API Requests must be made to API port.\n \tstatus code: 400, request id: , host id: " error.file="/go/src/github.com/vmware-tanzu/velero/pkg/persistence/object_store. go:191" error.function="github.com/vmware-tanzu/velero/pkg/persistence.(*objectBackupStore).IsValid" logSource="pkg/controlle r/backup_storage_location_controller.go:154"
time="2023-02-05T10:45:30Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location =velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:130"
time="2023-02-05T10:45:30Z" level=error msg="Error listing backups in backup store" backupLocation=velero/default controller= backup-sync error="rpc error: code = Unknown desc = InvalidArgument: S3 API Requests must be made to API port.\n\tstatus code : 400, request id: , host id: " error.file="/go/src/github.com/vmware-tanzu/velero-plugin-for-aws/velero-plugin-for-aws/objec t_store.go:308" error.function="main.(*ObjectStore).ListCommonPrefixes" logSource="pkg/controller/backup_sync_controller.go:1 07"
time="2023-02-05T10:45:30Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1, BackupStora geLocation \"default\" is unavailable: rpc error: code = Unknown desc = InvalidArgument: S3 API Requests must be made to API port.\n\tstatus code: 400, request id: , host id: )" controller=backup-storage-location logSource="pkg/controller/backup_stor age_location_controller.go:191"
AS I can see velero didn't manage to find the backup location as I provided the correct URL.
When I run
kubectl describe deployment -n velero velero
I found this output.
Name: velero
Namespace: velero
CreationTimestamp: Sun, 05 Feb 2023 12:45:24 +0200
Labels: component=velero
Annotations: deployment.kubernetes.io/revision: 1
Selector: deploy=velero
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: component=velero
deploy=velero
Annotations: prometheus.io/path: /metrics
prometheus.io/port: 8085
prometheus.io/scrape: true
Service Account: velero
Init Containers:
velero-velero-plugin-for-aws:
Image: velero/velero-plugin-for-aws:v1.0.0
Port: <none>
Host Port: <none>
Environment: <none>
Mounts:
/target from plugins (rw)
Containers:
velero:
Image: velero/velero:v1.10.1-rc.1
Port: 8085/TCP
Host Port: 0/TCP
Command:
/velero
Args:
server
--features=
--uploader-type=restic
Limits:
cpu: 1
memory: 512Mi
Requests:
cpu: 500m
memory: 128Mi
Environment:
VELERO_SCRATCH_DIR: /scratch
VELERO_NAMESPACE: (v1:metadata.namespace)
LD_LIBRARY_PATH: /plugins
GOOGLE_APPLICATION_CREDENTIALS: /credentials/cloud
AWS_SHARED_CREDENTIALS_FILE: /credentials/cloud
AZURE_CREDENTIALS_FILE: /credentials/cloud
ALIBABA_CLOUD_CREDENTIALS_FILE: /credentials/cloud
Mounts:
/credentials from cloud-credentials (rw)
/plugins from plugins (rw)
/scratch from scratch (rw)
Volumes:
plugins:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
scratch:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
cloud-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: cloud-credentials
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: velero-86f4984c96 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m45s deployment-controller Scaled up replica set velero-86f4984c96 to 1
Here all the Variables are set correctly but the volume path is not set. I am not sure where is issue basically.
This might help that when I look for velero backup location it is not set.
master-k8s#masterk8s-virtual-machine:~/velero-v1.2.0-darwin-amd64$ velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero Unavailable 2023-02-05 12:52:30 +0200 EET ReadWrite true
What I have done so far
I have try different blogs and stack question to fix. The issue is not resolved.
What is the main cause?
The main cause of this issue is I am not able to understand the logs. I want to know why velero not able to find the location in my case where as in different blogs the same setup works.
What I want to do Or how can you help me?
Please help me to find the main issue. Why my backup fails with error ,
master-k8s#masterk8s-virtual-machine:~/velero-v1.2.0-darwin-amd64$ velero backup create mytest --inc lude-namespaces postgres-operator
Backup request "mytest" submitted successfully.
Run `velero backup describe mytest` or `velero backup logs mytest` for more details.
master-k8s#masterk8s-virtual-machine:~/velero-v1.2.0-darwin-amd64$ velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION S ELECTOR
mytest Failed 0 0 2023-02-05 13:00:03 +0200 EET 29d default < none>
master-k8s#masterk8s-virtual-machine:~/velero-v1.2.0-darwin-amd64$ velero backup logs mytest
An error occurred: gzip: invalid header
I will be really thanks full for your help and support in advance.