I am using ubuntu 16.04 and virtualenvwrapper for the virtual environment. For production, I want to run automatically Airflow scheduler with Systemd.
How can I configure Airflow scheduler in a virtual environment and within Systemd?
There are example systemd configuration files for each airflow component in https://github.com/apache/incubator-airflow/tree/1.9.0/scripts/systemd. Then just make sure virtualenv path for airflow, for example, ExecStart=/my/project/venv/bin/airflow scheduler. If your airflow home directory is different from the default (~/airflow), you can override that with Environment="AIRFLOW_HOME=/my/airflow.
Related
I have a rancher installation on cloud (integrated with harvester) and a couple of VM's in a local node (with K3os), created with harvester.
Now I would like to connect the K3S cluster running on a VM with rancher, but when I try to run in the VM the script of agent given to me by rancher, it goes into an error:
systemctl: command not found
Am I doing something wrong?
I found the problem.
When you run a VM with k3os, a k3s cluster within it is also started in the VM as mentioned before. So I was wrong choosing "Create a cluster", i should have chosen instead "import a cluster". In this way, the script you run into the VM works perfectly.
In a Jupyter notebook connected to a GCP Spark cluster, the cell !pip3 install pyLDAvis==3.2.1 works, but gives a warning:
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager.
It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
The warning is not unique to pyLDAvis, other packages — even numpy — give the same warning.
Running the notebook as root shouldn't be the default. How can the default user in the notebook be set to singhj rather than root? I have searched through IPython Configuration and customization for any hints.
Configuration: Fresh cluster in GCP Dataproc, default Jupyter notebook, nothing customized.
The Jupyter server in a Dataproc cluster is run by the systemd service defined in the file /usr/lib/systemd/system/jupyter.service.
If you want to change the user it runs as, then you can modify that file and replace the line saying User=root with one saying the name of the user you want (e.g. User=singhj in your example).
Then, once the file has been updated, restart the systemd service by running the following commands as root:
systemctl daemon-reload
systemctl restart jupyter
If you want to automate that, you can write an initialization action to make the change at cluster creation time.
I am using Canonical's Multipass on a Raspberry Pi with LXD driver to start virtual machines (VMs). Given my configuration, I need to view real-time changes/updates happening in the virtual machine on the host, i.e. externally from the virtual machine. The current way I can do this is by unmounting and mounting the VM image file (.img file) from time to time which is not ideal.
Is there a way of doing a real-time follow-up on the changes with the .img file?
After reaching out to the LXD project team (thanks Thomas Parrott
) the best way is not to mount the .img file but to use lxc. Given how Multipass is installed using the LXD driver we can access the VM with lxc exec <instance> --project multipass -- <cmd> <args>.
I want to start practicing with k8s for the CKAD exam. I run on ubuntu 18.04.
I noticed everywhere that I need to download Virtualbox for minikube. I believe that VB is needed in case I don't start my cluster with a driver but if I use the Docker driver when I start my cluster shouldn't that be enough? Is microk8s a better option?
It seems that the preferred way is use --driver=docker driver instead of --driver=none for minikube, although it is technically not baremetal as it is significantly easier to configure and does not require root access. The ‘none’ driver is recommended for advanced users only. (info below from https://minikube.sigs.k8s.io/docs/drivers/docker/)
docker
Overview
The Docker driver allows you to install Kubernetes into an existing Docker install. On Linux, this does not require virtualization to be enabled.
Requirements
Install Docker 18.09 or higher
amd64 or arm64 system.
Usage
Start a cluster using the docker driver:
minikube start --driver=docker
To make docker the default driver:
minikube config set driver docker
Yes you can. Check here.
Minikube also supports a --driver=none option that runs the Kubernetes components on the host and not in a VM. Using this driver requires Docker and a Linux environment but not a hypervisor.
Jus run
$ minikube start
Caution: If you use the none driver, some Kubernetes components run as privileged containers that have side effects outside of the Minikube environment. Those side effects mean that the none driver is not recommended for personal workstations
I want to run simple shell script when lxc container is stopped.
I try to use
lxc.hook.post-stop = script I want to execute
in my lxc container config file,
but the script is not being invoked when the lxc container is stopped.
Is there any other way to achieve the same.
my system details:
Ubuntu : 16.04 LTS,
LXC version : 2.0.8
the shell script contains:
#!/bin/bash
##Start the container if it is stopped
lxc-start -n nameofthecontainer
My intention is to keep the container running all the time, hence looking for some notification when the container is stopped so that I can restart.
Thank you for the help.
Verified with latest LXC version 3.0.1 and hooks are working as expected. The issue was older LXC version