Looking at this example of mounting an EFS volume for persisting docker volumes in ECS, I'm unsure how to provide the correct mount point for the availability zone that the instance is in. I have two availability zones in my stack and need the correct mount point to insert in this section of the cfn-init:
01_mount:
command: !Join [ "", [ "mount -t nfs4 -o nfsvers=4.1 ", !ImportValue '!Ref FileSystem', ".efs.", !Ref 'AWS::Region', ".amazonaws.com:/ /", !Ref MountPoint ] ]
02_fstab:
command: !Join [ "", [ "echo \"", !ImportValue '!Ref FileSystem', ".efs.", !Ref 'AWS::Region', ".amazonaws.com:/ /", !Ref MountPoint, " nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0\" >> /etc/fstab" ] ]
03_permissions:
command: !Sub "chown -R ec2-user:ec2-user /${MountPoint}"
It is no longer necessary to use the availability-zone-specific mount target when mounting an EFS filesystem, if you are using the DNS settings in your VPC and have other necessary prerequisites in place in the VPC configuration.
File system DNS name – Using the file system's DNS name is your simplest mounting option. The file system DNS name will automatically resolve to the mount target’s IP address in the Availability Zone of the connecting Amazon EC2 instance. You can get this DNS name from the console, or if you have the file system ID, you can construct it using the following convention:
file-system-id.efs.aws-region.amazonaws.com
(emphasis added)
http://docs.aws.amazon.com/efs/latest/ug/mounting-fs-mount-cmd-dns-name.html
This feature was introduced in December, 2016, several months after the service was released from preview. Formerly, the hostname style shown above had to be prepended with the availability zone you wanted. That option is still supported, but this option effectively eliminates this awkward configuration requirement, both in docker and on ordinary instances with fstab mounts.
See the referenced page for the VPC configuration elements that must be in place for this solution to work in your VPC.
Related
Trying to launch a Fargate task that uses an EFS Volume.
When starting the task from ECS Console, I'm getting this error :
ResourceInitializationError: failed to invoke EFS utils commands to set up EFS volumes: stderr: Failed to resolve "fs-019a4b2d1774c5586.efs.eu-west-1.amazonaws.com" - check that your file system ID is correct, and ensure that the VPC has an EFS mount target for this file system ID. See https://docs.aws.amazon.com/console/efs/mount-dns-name for more detail. Attempting to lookup mount target ip address using botocore. Failed to import necessary dependency botocore, please install botocore first. : unsuccessful EFS utils command execution; code: 1
File system Id is correct. I've mounted the volume from an ec2 instance in the same VPC, all good.
Following steps defined here : https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/?nc1=h_ls
I cannot figure out where to specify outbound rule for ECS service or task. See image
Thanks in advance.
As #MarkB stated, i've edited the outbound rule and added the port 2049 (NFS) to the EFS security group, and it's workin fine.
Basically the ECS'S security group should allow ssh in the ingress and nfs protocol on the port 2049 to the Securitygroup of the mount target and
Mount target's security group should allow nfs protocol on the 2049 port.
I set up a local Kubernetes cluster using Kind, and then I run Apache-Airflow on it using Helm.
To actually create the pods and run Airflow, I use the command:
helm upgrade -f k8s/values.yaml airflow bitnami/airflow
which uses the chart airflow from the bitnami/airflow repo, and "feeds" it with the configuration of values.yaml.
The file values.yaml looks something like:
web:
extraVolumeMounts:
- name: functions
mountPath: /dir/functions/
extraVolumes:
- name: functions
hostPath:
path: /dir/functions/
type: Directory
where web is one component of Airflow (and one of the pods on my setup), and the directory /dir/functions/ is successfully mapped from the cluster inside the pod. However, I fail to do the same for a single, specific file, instead of a whole directory.
Does anyone knows the syntax for that? Or have an idea for an alternative way to map the file into the pod (its whole directory is successfully mapped into the cluster)?
There is a File type for hostPath which should behave like you desire, as it states in the docs:
File: A file must exist at the given path
which you can then use with the precise file path in mountPath. Example:
web:
extraVolumeMounts:
- name: singlefile
mountPath: /path/to/mount/the/file.txt
extraVolumes:
- name: singlefile
hostPath:
path: /path/on/the/host/to/the/file.txt
type: File
Or if it's not a problem, you could mount the whole directory containing it at the expected path.
With this said, I want to point out that using hostPath is (almost always) never a good idea.
If you have a cluster with more than one node, saying that your Pod is mounting an hostPath doesn't restrict it to run on a specific host (even tho you can enforce it with nodeSelectors and so on) which means that if the Pod starts on a different node, it may behave differently, not finding the directory and / or file it was expecting.
But even if you restrict the application to run on a specific node, you need to be ok with the idea that, if such node becomes unavailable, the Pod will not be scheduled on its own somewhere else.. meaning you'll need manual intervention to recover from a single node failure (unless the application is multi-instance and can resist one instance going down)
To conclude:
if you want to mount a path on a particular host, for whatever reason, I would go for local volumes.. or at least use hostPath and restrict the Pod to run on the specific node it needs to run on.
if you want to mount small, textual files, you could consider mounting them from ConfigMaps
if you want to configure an application, providing a set of files at a certain path when the app starts, you could go for an init container which prepares files for the main container in an emptyDir volume
I try to set up graylog.
This deploy works. But i need to add a volume to graylog deploy. Because I want to install plugins.
When i add volume (hostPath) and start pod, I get an error in my pod:
ERROR StatusLogger File not found in file system or classpath: /usr/share/graylog/data/config/log4j2.xml
ERROR StatusLogger Reconfiguration failed: No configuration found for '70dea4e' at 'null' in 'null'
06:37:19.707 [main] ERROR org.graylog2.bootstrap.CmdLineTool - Couldn't load configuration: Properties file /usr/share/graylog/data/config/graylog.conf doesn't exist!
I see, that pod create catalogues (owner id 1100:1100) in volume, but there is no any file there.
Kubernetes version is 1.20
Runtime in my kubernetes cluster is "containerd".
My Graylog dеploy:
Volume ounys for container:
volumeMounts:
- mountPath: /usr/share/graylog/data
name: graylog-data
Volume:
volumes:
- name: graylog-data
hostPath:
path: /mnt/k8s-storage/graylog-data
type: DirectoryOrCreate
There are few things to look at here starting from the concept of the hostPath volume:
A hostPath volume mounts a file or directory from the host node's
filesystem into your Pod.
Pods with identical configuration (such as created from a PodTemplate)
may behave differently on different nodes due to different files on
the nodes
The files or directories created on the underlying hosts are only
writable by root. You either need to run your process as root in a
privileged Container or modify the file permissions on the host to be
able to write to a hostPath volume
The hostPath would be good if for example you would like to use it for log collector running in a DaemonSet but in your described use case it might not be ideal because you don't directly control which node your pods will run on, so you're not guaranteed that the pod will actually be scheduled on the node that has the data volume.
But if that not the case, you also need to notice that type: DirectoryOrCreate is not best here as I see that you want to load a file. It would be better to use either:
File: A file must exist at the given path
or
FileOrCreate: If nothing exists at the given path, an empty file will be created there as needed with permission set to 0644, having the same group and ownership with Kubelet.
Lastly, there might be a permissions problem. As already stated:
The files or directories created on the underlying hosts are only
writable by root.
Graylog is running with the userid 1100 which might cause a permission denial. Also, I have found a similar issue that might be helpful for you.
I'm using a terastation's iscsi capabilities to create storage for my k8s 1.14 cluster (kubeadm, ubuntu 18.04). I check the iqn:
iscsiadm --mode node
192.168.2.113:3260,1 iqn.2004-08.jp.buffalo.7403bd2a30a0.drupal-mysql
There's no ":". when I try to use
volumes:
- name: iscsi-data
iscsi:
targetPortal: 192.168.2.113:3260
iqn: "iqn.2004-08.jp.buffalo.7403bd2a30a0.drupal-mysql"
lun: 0
fsType: xfs
i get the error:
spec.template.spec.volumes[0].iscsi.iqn: Invalid value: "iqn.2004-08.jp.buffalo.7403bd2a30a0.drupal-mysql": must be valid format
i know its looking for something that ends in a ":name" but I can't figure out what thats supposed to be for the life of me. I know the iscsi drive mounts because i can see it on my node and was able to format it using xfs. I think I'm missing something really simple.
Thanks
iSCSI network storage standard is fully documented in RFC 3720 and RFC 3721 with an appropriate IQN constructing format for iSCSI names.
iSCSI qualified name (IQN), coressponds to the following form:
iqn.yyyy-mm.naming-authority:unique-name, where:
iqn – the prefix iqn.
yyyy-mm – the year and month when the naming authority was established. For example: 1992-08.
naming-authority – the organizational naming authority string, usually reverse syntax of the Internet domain name of the naming
authority. For example: com.vmware.
unique name – any name you want to use, such as the name of your host. For example: host-1
In the above k8s volume spec case you might try to specify IQN like:
iqn: "iqn.2004-08.jp.buffalo:7403bd2a30a0.drupal-mysql"
Find some relative example about iSCSI volume provisioning in k8s cluster here.
Using Rook 0.9.3 I was able to bring up a Ceph-based directory for a MySQL-database on a three-node Kubernetes cluster (1 master, two workers) simply as follows:
kubectl create -f cluster/examples/kubernetes/ceph/operator.yaml
kubeclt create -f cluster/examples/kubernetes/ceph/cluster.yaml
vim cluster/examples/kubernetes/ceph/storageclass.yaml # change xfs to ext4
kubectl create -f cluster/examples/kubernetes/ceph/storageclass.yaml
kubectl create -f cluster/examples/kubernetes/mysql.yaml
When I now bash into the pod wordpress-mysql-* I can see that /var/lib/mysql is mounted from /dev/rbd1. If I create a random file in this directory and then delete the pod, the file has persisted when a new instance of the pod comes up.
My first worker contains these directories in /var/lib/rook: mon-a mon-c mon-d osd0 rook-ceph. My second worker contains only one directory in /var/lib/rook: mon-b. This and other evidence (from df) suggest that Rook (and by extension Ceph) stores all of its file data (e.g. all blocks that constitute the mounted /var/lib/mysql) in /var/lib/rook/osd0, i.e. once on a single node.
I would have expected that blocks are distributed across several nodes so that when one node (the first worker, in my case) fails, data access is still available. Is this a naive expectation? If not, how can I configure Rook accordingly? Also, I have second, unformatted disks on both worker nodes, and I would prefer for Rook/Ceph to use those. How can this be accomplished?
for using other partition as osd
you should change cluster.yml and add
nodes:
- name: "kube-node1"
devices:
- name: "sdb"
- name: "kube-node2"
devices:
- name: "sdb"
- name: "kube-node3"
devices:
- name: "sdb"