I am trying to set up a backup plan for Postgres using the following Protected application configuration ( https://cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/how-to/protected-application#backup-one-restore-all )
kind: ProtectedApplication
apiVersion: gkebackup.gke.io/v1alpha2
metadata:
name: pg-backup
namespace: default
spec:
resourceSelection:
type: Selector
selector:
matchLabels:
app: pg
components:
- name: pg-backup
resourceKind: StatefulSet
resourceNames: ["postgres-postgresql"]
strategy:
type: BackupOneRestoreAll
backupOneRestoreAll:
backupTargetName: "postgres-postgresql"
backupPostHooks:
- command:
- /sbin/fsfreeze
- --unfreeze
- /var/log/postgresql
container: postgresql
name: unquiesce
backupPreHooks:
- command:
- /sbin/fsfreeze
- --freeze
- /var/log/postgresql
container: postgresql
name: quiesce
volumeSelector:
matchLabels:
volume-type: primary
I get an exception in GKE saying that
ailed to get selected PVCs for the backup Pod default/postgres-postgresql-1: failed to get resource /default/v1/PersistentVolumeClaim/data-postgres-postgresql-1: resource not found
But I already have the PVC Present and the labels are also added to the PVC's. Is there anything that is being missed here?
Related
According to Kubernetes documentation, we should specify type: DirectoryOrCreate if we want to create directory on the host. The default option is "no checks will be performed before mounting the hostPath volume".
However, I am seeing directory gets created on host even when no type is specified:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-user-hostpath
spec:
replicas: 1
selector:
matchLabels:
app: busybox-user-local-storage1
template:
metadata:
labels:
app: busybox-user-local-storage1
spec:
containers:
- name: busybox
image: busybox:latest
command: ["/bin/sh", "-ec", "while :; do echo $(date '+%Y-%m-%d %H:%M:%S') deployment1 >> /home/test.txt; sleep 5 ; done"]
volumeMounts:
- name: busybox-hostpath
mountPath: /home
volumes:
- name: busybox-hostpath
hostPath:
path: /home/maintainer/data
/home/maintainer/data directory did not exist before running the pod. After deployment, I can see the directory is created. This goes against the documentation unless I am missing something. I was expecting the pod should crash but I can see the files are created. Any idea please?
This is something that goes back in time, before type was even implemented for hostPath volume. When unset should just go and default directly to create an empty directory, and it's a backward compatible implementation, because no one had the option to add type and forcing an error when it's not defined would have broken all previously created pods without it. You can take a look into the actual design-proposal: https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-hostpath-qualifiers.md#host-volume
The design proposal clearly specifies that "unset - If nothing exists at the given path, an empty directory will be created there. Otherwise, behaves like exists"
I'm trying to setup GCR with kubernetes
and getting Error: ErrImagePull
Failed to pull image "eu.gcr.io/xxx/nodejs": rpc error: code = Unknown desc = Error response from daemon: pull access denied for eu.gcr.io/xxx/nodejs, repository does not exist or may require 'docker login'
Although I have setup the secret correctly in the service account, and added image pull secrets in the deployment spec
deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert
kompose.version: 1.18.0 (06a2e56)
creationTimestamp: null
labels:
io.kompose.service: nodejs
name: nodejs
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: nodejs
spec:
containers:
- env:
- name: MONGO_DB
valueFrom:
configMapKeyRef:
key: MONGO_DB
name: nodejs-env
- name: MONGO_HOSTNAME
value: db
- name: MONGO_PASSWORD
valueFrom:
secretKeyRef:
name: mongo-secret
key: MONGO_PASSWORD
- name: MONGO_PORT
valueFrom:
configMapKeyRef:
key: MONGO_PORT
name: nodejs-env
- name: MONGO_USERNAME
valueFrom:
secretKeyRef:
name: mongo-secret
key: MONGO_USERNAME
image: "eu.gcr.io/xxx/nodejs"
name: nodejs
imagePullPolicy: Always
ports:
- containerPort: 8080
resources: {}
imagePullSecrets:
- name: gcr-json-key
initContainers:
- name: init-db
image: busybox
command: ['sh', '-c', 'until nc -z db:27017; do echo waiting for db; sleep 2; done;']
restartPolicy: Always
status: {}
used this to add the secret, and it said created
kubectl create secret docker-registry gcr-json-key --docker-server=eu.gcr.io --docker-username=_json_key --docker-password="$(cat mycreds.json)" --docker-email=mygcpemail#gmail.com
How can I debug this, any ideas are welcome!
It looks like the issue is caused by lack of permission on the related service account
XXXXXXXXXXX-compute#XXXXXX.gserviceaccount.com which is missing Editor role.
Also,we need to restrict the scope to assign permissions only to push and pull images from google kubernetes engine, this account will need storage admin view permission which can be assigned by following the instructions mentioned in this article [1].
Additionally, to set the read-write storage scope when creating a Google Kubernetes Engine cluster, use the --scopes option to mention this scope "storage-rw"[2].
[1] https://cloud.google.com/container-registry/docs/access-control
[2]https://cloud.google.com/container-registry/docs/using-with-google-cloud-platform#google-kubernetes-engine”
If the VM instance for pushing or pulling images and the Container Registry storage bucket are in the same Google Cloud Platform project, the Compute Engine default service account is configured with appropriate permissions to push or pull images.
If the VM instance is in a different project or if the instance uses a different service account, you must configure access to the storage bucket used by the repository.
By default, a Compute Engine VM has the read-only access scope configured for storage buckets. To push private Docker images, your instance must have read-write storage access scope configured as described in Access scopes.
Please have 1 for further reference:
Please follow below table as 2:
Action Permission Role Role Title
Pull (Read Only) - storage.objects.get roles/storage.objectViewer Storage Object Viewer
storage.objects.list
Also, you could share if there having any error code as you are having trouble in any steps.
I'm attempting to inject a ReplicationController's randomly generated pod ID extension (i.e. multiverse-{replicaID}) into a container's environment variables. I could manually get the hostname and extract it from there, but I'd prefer if I didn't have to add the special case into the script running inside the container, due to compatibility reasons.
If a pod is named multiverse-nffj1, INSTANCE_ID should equal nffj1. I've scoured the docs and found nothing.
apiVersion: v1
kind: ReplicationController
metadata:
name: multiverse
spec:
replicas: 3
template:
spec:
containers:
- env:
- name: INSTANCE_ID
value: $(replicaID)
I've tried adding a command into the controller's template configuration to create the environment variable from the hostname, but couldn't figure out how to make that environment variable available to the running script.
Is there a variable I'm missing, or does this feature not exist? If it doesn't, does anyone have any ideas on how to make this to work without editing the script inside of the container?
There is an answer provided by Anton Kostenko about inserting DB credentials into container environment variables, but it could be applied to your case also. It is all about the content of the InitContainer spec.
You can use InitContainer to get the hash from the container’s hostname and put it to the file on the shared volume that you mount to the container.
In this example InitContainer put the Pod name into the INSTANCE_ID environment variable, but you can modify it according to your needs:
Create the init.yaml file with the content:
apiVersion: v1
kind: Pod
metadata:
name: init-test
spec:
containers:
- name: init-test
image: ubuntu
args: [bash, -c, 'source /data/config && echo $INSTANCE_ID && while true ; do sleep 1000; done ']
volumeMounts:
- name: config-data
mountPath: /data
initContainers:
- name: init-init
image: busybox
command: ["sh","-c","echo -n INSTANCE_ID=$(hostname) > /data/config"]
volumeMounts:
- name: config-data
mountPath: /data
volumes:
- name: config-data
emptyDir: {}
Create the pod using following command:
kubectl create -f init.yaml
Check if Pod initialization is done and is Running:
kubectl get pod init-test
Check the logs to see the results of this example configuration:
$ kubectl logs init-test
init-test
I noticed that mysql 5.7 images on google container engine terminates itself after every 2 to 3 weeks of running in my cluster . i configured a small cluster as a test environment . I have 3 nodes with one for database , one for api and the other for my node js front end .
This all works well after my configuration i am able to create my database and its accompanying tables , stored procedures and our usual db objects . My back ends all connects to the db and also my front ends are all up and running . Then suddenly after a period i can estimate about 3 weeks my back ends can no longer connect to my databases any more . it just points out that it cant connect to mysql server . I dash to my cmd and check if the mysql pod is running . it actually is running . But i cant connect access my db . I had to redeploy the mysql image luckily because of my persistent volumes could still recover the db files . The second time it occurred it kept saying no root user , i was surprised because i normally do all my db design and all using this user . The third time it just couldn't locate my db any more . I'm also thinking it might be my deployment script i attached it here as well for nay suggestions :
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: mysql
labels:
app: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.7
args:
- "--ignore-db-dir=lost+found"
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
This is what i get in the logs
W1231 11:59:23.713916 14792 cmd.go:392] log is DEPRECATED and will be
removed in a future version. Use logs instead.
Initializing database
2017-12-31T10:57:23.236067Z 0 [Warning] TIMESTAMP with implicit DEFAULT
value is deprecated. Please use --explicit_defaults_for_timestamp server
option (see documentation for more details).
2017-12-31T10:57:23.237652Z 0 [ERROR] --initialize specified but the data
directory has files in it. Aborting.
2017-12-31T10:57:23.237792Z 0 [ERROR] Aborting
I'm having a tough time connecting to a Cloud SQL Instance from a Java App running in a Google Container Engine Instance.
I whitelisted the external instance IP from the Access Control of CloudSQL. Connecting from my local machine works well, however I haven't managed to establish a connection from my App yet.
I'm configuring the Container as (cloud-deployment.yaml):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: APPNAME
spec:
replicas: 1
template:
metadata:
labels:
app: APPNAME
spec:
imagePullSecrets:
- name: APPNAME.com
containers:
- image: index.docker.io/SOMEUSER/APPNAME:latest
name: web
env:
- name: MYQL_ENV_DB_HOST
value: 111.111.111.111 # the cloud sql instance ip
- name: MYQL_ENV_MYSQL_PASSWORD
value: THEPASSWORD
- name: MYQL_ENV_MYSQL_USER
value: THEUSER
ports:
- containerPort: 9000
name: APPNAME
using the connection url jdbc:mysql://111.111.111.111:3306/databaseName, resulting in:
Error while executing: Access denied for user 'root'#'ip adress of the instance' (using password: YES)`
I can confirm that the Container Engine external IP is set on the SQL instance.
I don't want to use the Cloud Proxy Image for now as I'm still in development stage.
Any help is greatly appreciated.
You must use the cloud SQL proxy as described here: https://github.com/GoogleCloudPlatform/cloudsql-proxy/blob/master/README.md