Google cloud after VM Import no free space available on root drive - import

I created a Postgres server locally using virualbox using Ubuntu 16.04. Using the import tool to move it to Google cloud seemed to work fine, but the root drive shows 100% full. None of the disk expansion instructions (including creating a snapshot and recreating the boot drive) seem to make any space available.
There seems to be a boot drive and a root drive. But the root drive shows it is all used. The boot drive shows space available, but it should be 15G in size not 720M.
Filesystem Size Used Avail Use% Mounted on
udev 1.8G 0 1.8G 0% /dev
tmpfs 370M 5.3M 365M 2% /run
/dev/mapper/techredo--vg-root 2.5G 2.5G 0 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sdb1 720M 121M 563M 18% /boot
tmpfs 370M 0 370M 0% /run/user/406485188

I checked if is possible to use LVM in GCP instances, and I find out that you're free to use it but is not supported by Google Cloud since instances doesn't use LVM by default.
On the other hand, you need to make sure that the Linux Guest Environment is installed in your instance, so you can get the automatic resizing feature. Please follow this guide to learn how to validate: https://cloud.google.com/compute/docs/images/install-guest-environment#wgei
Since your root partition is full and you're not able to install more programs I suggest you 2 workarounds:
Workaround 1: Create a new VirtualBox VM with and import it again, please note that your root partition is pretty small (2.5G) so I suggest you next time create a partition with at least 10GB, and avoid use LVM during the installation.
After your instance is ready in GCP, please check if the Linux Guest Environment is installed in your instance, if not install it: https://cloud.google.com/compute/docs/images/install-guest-environment
Workaround 2: Check which directory is causing problems and then which files are consuming your disk space, delete them to gain space, install the Guest enviroment and try to resize your instance.
a) To check the directories and files sizes, follow these steps:
There are several tools that can display your disk usage graphically but since your root partition is full you'll have to get the information by running commands (old school style).
Please follow these steps:
Please go to the root directory:
cd /
Please run this command to get the size of the main subdirectories under the root partition:
sudo du -aBM -d 1 . -R | sort -nr | head -20
NOTE: Identify which directory is eating your root partition.
Please run this command to get a full list of the files and its sizes:
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh
NOTE: The above command will display all files and directories too fast, so in order to scroll down slowly, please run the same command adding the "less" instruction:
du -k *| sort -nr | cut -f2 | xargs -d '\n' du -sh |less
Press the spacebar to scroll down.
Please keep in mind that you have to go to the directory you want to analyze before running the commands in step 3 or 4 (just in case you want to analyze another directory).
Additional to this you can run the command "apt-get clean" to clear the downloaded packages (.deb files) that usually consumes a high part of your disk.
b) To resize your instance, you have 2 options:
Resize your VM instance "primary-server" by following this guide[1].
NOTE: The steps included in this option are pretty easy to follow, if this doesn't work try the second option which requires advanced Linux abilities.
Create a snapshot from the VM instance "primary-server".
2.1 Create a new instance based on a Linux distribution.
2.2 Once it's been created, stop the instance.
2.3 Follow this guide to add an additional disk[2].
NOTE: Basically you have to edit the instance "primary-server" and add an additional disk, don't forget to select the snapshot option from the "Source type" list and click on the snapshot you just created.
2.4 Start the instance.
2.5 Mount the disk by following this guide[3].
NOTE: Please skip step 4. The additional disk it's actually a boot disk so it's been already formatted. So, don't apply the format to it; just mount it, .
2.6 Check the permissions of the file "/etc/fstab".
NOTE: The permissions should be "-rw-r--r--" and the owner "root"
2.6.1 Delete files to reduce the disk size.
2.7 Unmount the disk at OS level.
2.8 Stop the instance.
2.9 Detach the additional disk from the new instance in GCP.
NOTE: Please follow this guide[4] and instead of clicking on X next to the boot disk, please click on X next to the additional disk.
2.10 Create a new instance and instead of using an image in the "boot disk" section, please use the disk you just restored.
NOTE: For this please go to the "Boot disk" section and click on the "Change" button, then go to the "Existing" TAB and select the disk you just restored.
REFERENCES:
[1] https://cloud.google.com/compute/docs/disks/add-persistent-disk#inaccessible_instance
[2] https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk
[3] https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting
[4] https://cloud.google.com/compute/docs/disks/detach-reattach-boot-disk#detach_disk
Please let me know the results.

Related

How to create a permanent login from jupyter notebook to github with ssh_rsa key pair

So my question is this,
When creating a notebook in Sagemaker AWS I need to help the devEngineer keep his secret key in .ssh/id_rsa as the file after every instance reboot becomes empty.
He requires a github repo to be downloaded and he has to work on the code and then push the updates as needed.
Please let me know what details I need to provide to help you help me.
Thanks.
This is the filesystems for my notebook instance:
sh-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 16G 76K 16G 1% /dev
tmpfs 16G 0 16G 0% /dev/shm
/dev/nvme0n1p1 94G 76G 19G 81% /
/dev/nvme1n1 99G 40G 55G 43% /home/ec2-user/SageMaker
Note that one pointing to /home/ec2-user/SageMaker is the only one which is saved between reboots. Since ssh keys are stored in /home/ec2-user/.ssh, they are lost after reboot.
The way I make it work is:
Create the folder /home/ec2-user/SageMaker/.ssh
Run ssh-keygen and set the location /home/ec2-user/SageMaker/.ssh/id_rsa
Clone repo with GIT_SSH_COMMAND="ssh -i ~/SageMaker/.ssh/id_rsa -F /dev/null" git clone git#domain:account/repo.git
cd repo
Set your repo to use the custom location with git config core.sshCommand "ssh -i ~/SageMaker/.ssh/id_rsa -F /dev/null"
Based on https://superuser.com/a/912281
Thanks for using Amazon SageMaker!
SageMaker Notebook Instance had launched a new feature few months ago which allows you to associate git repositories with your Notebook Instances.
Please take a look at this blog post and documentation on how to use this feature. Let us know if this works for your use case.
Thanks,
Neelam

Kickstart: create lvm volume group without partition

I am trying to setup a redhat 7 kickstart for a server with 2 disks.
On the second disk I want to use the full disk without partitioning in lvm.
Once the system is installed, the config works:
pvcreate /dev/sdb
vgcreate data /dev/sdb
lvcreate -l +100%FREE -n data data
mkfs.xfs /dev/mapper/data-data
echo -e "/dev/mapper/data-data\t/data\txfs\tdefaults\t0 1" >> /etc/fstab
mount /data
But I cannot manage to have the following partitioning to work as expected
The partitioning system on kickstart as I know it will only create a partition on /dev/sdb and the volume is create on /dev/sdb1 at the end.
I managed to workaround the issue by using a postscript, but I compiled packages to install in this folder, so I would need the formating to be done before at least in a pre script if the partitioning is not possible.

How to Mount Disk for Google Cloud Compute Engine to use with /home?

I have a VM Instance with a small 10GB boot disk running CentOS 7 and would like to mount a larger 200GB Persistent Disk to contain data relating to the /home directory from a previous dedicated server (likely via scp).
Here's what I tried:
Attempt #1, Symlinks Might work, but some questions.
mounted the disk to /mnt/disks/my-persistent-disk
created folders on the persistent disk that mirror the folders in the old server's /home directory.
created a symlink in the /home directory for each folder, pointing to the persistent disk.
scp from old server to the VM /home/example_account for the first account. Realized scp does not follow symlinks (oops) and therefore the files went to the boot drive instead of the disk.
I suppose I could scp to /mnt/disks/my-persistent-disk and manage the symlinks and folders. Would this pose a problem? Would making an image of the VM with this configuration carry over to new instances (with autoscaling etc)?
Attempt #2, Mounting into /home.
Looking for a more 'natural' configuration that works with ftp, scp etc, I mounted the disk in /home/example_account
$ sudo mkdir -p /home/example_account
$ sudo mount -o discard,defaults /dev/sdc /home/example_account
$ sudo chmod a+w /home/example_account
#set the UUID for mounting at startup
$ sudo blkid /dev/sdc
$ sudo nano /etc/fstab
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /
sdc 8:32 0 200G 0 disk /home/example_account
scp from old server to the VM in the /home/example_account works fine. Yay. However, I would like to have more than just 1 folder in the /home directory. I suppose I could partition the disk but this feels a bit cumbersome and I'm not exactly sure how many accounts I will use in the future.
Attempt #3, Mount as /home
I felt the best solution was to have the persistent disk mount as the /home directory. This would allow for easily adding new accounts within /home without symlinks or disk partitions.
Attempted to move /home directory to /home.old but realized the Google Cloud Compute Engine would not allow it since I was logged into the system.
Changed to root user, but still said myusername#instance was logged in and using the /home directory. As root, I issued pkill -KILL -u myusername and the SSH terminated - apparently how the Google Cloud Compute Engine works with their SSH windows.
As I cannot change the /home directory, this method does not seem viable unless there is a workaround.
My thoughts:
Ideally, I think #3 is the best solution but perhaps there is something I'm missing (#4 solution) or one of the above situations is the preferable idea but perhaps with better execution.
My question:
In short, how to I move an old server's data to a Google Cloud VM with a persistent disk?

Limit Disk usage in Docker+MongoDB

I am using the official mongo Docker image to start a MongoDB container where my boot disk is limited (e.g. 10G) I configured the docker to run with Google Cloud Logging driver and was hoping Google to store all the logs and save my local disk space. However, I notice the disk continues to grow:
$ df -h
/dev/sda1 9.9G 4.5G 4.9G 49%
As I digged deeper I realized the size of docker containers seems to be growing over time.
$ sudo du -sh /var/lib/docker/
3.6G /var/lib/docker/
However, I can't go further as somehow I can't access the directories within.
If I go inside the docker and du -sh the root, I don't find any suspicious directories occupying space.
So my problem is how do I find out where the disk space is used and how do I eliminate it.
My docker startup command (shown without project options)
docker run -d --log-driver=gcplogs mongo mongod
EDIT: I noticed the size growing has stopped at 4.5GB from ~3GB for a while. So I supposed it has reached some equilibrium now.

installations disappearing on google compute engine

I'm experiencing some weird behavior on the google compute engine. I made a new instance with ubuntu on it. I installed a node app I'm working on, pulled code from github etc...
then I installed mongodb and nginx. The weird thing is, every time I leave the session, and reconnect, my mongodb and nginx installation files disappear.
for example, when I install nginx I find the nginx installation on /etc/nginx where I can find like nginx.conf. but when I left the compute engine console session, and reconnected later, that directory was gone. same thing is happening with mongodb.
my node installation under /home/abdul/mystuff doesn't disappear though.
is this normal? is it a setting?
details:
this is an ubuntu image (idk which version, and not sure how to check)
using the following to install nginx:
sudo apt-get update
sudo apt-get install nginx
result of command
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
└─sda1 8:1 0 10G 0 part /var/lib/docker/aufs
sdb 8:16 0 5G 0 disk
└─sdb1 8:17 0 5G 0 part /home
Looks like you're running a Docker container on your instance (/var/lib/docker/aufs) and installing the software inside the container.
If you want to save changes back to the image, it is possible to use the docker commit command, but this is almost definitely not what you want.
Instead, use a Dockerfile to build images and update it whenever you want to make a change. This way you can easily recreate the image and make changes without starting from scratch. For persistence (.e.g. config files and databases) use volumes, which are just directories stored outside of the Union File System as normal directories on the host.